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(57)Abstract: 

PROBLEM TO BE SOLVED: To perform retrieval by giving 
structure information to part of a retrieval key even if 
information regarding a document structure is not known by 
deciding the similarity of a composite media document according 
to feature information extracted from a constituent element of 
the composite media document and feature information 
extracted from a constituent element of a retrieval condition. 
SOLUTION: The syntax of the given composite multimedia 
document is analyzed (S1) and 1st feature information is 
extracted from a constituent element of the composite media 
document (S2). The document and extracted feature information 
are stored (S3) and the composite media document is inputted 
as a retrieval condition (S4). The syntax of the retrieval 
condition is analyzed (S5) and 2nd feature information is 
extracted from a constituent element of the retrieval condition 
obtained as a result of the analysis (S6). Then the similarity 
between two composite media documents is decided (S7) 
according to the 1st feature information and 2nd feature 
information which are stored and similar composite media 
documents are outputted (S8). 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 

[Claim(s)] 

[Claim 1]ln a similar retrieval method of a compound media document which is a structured 
document which comprises speech information which adds to text information, picture 
information, and voice data that is data of human being's voice, and contains composition data 
containing CD and a record and music data, The 1st characteristic information is extracted 
from a component of said compound media document which analyzes the syntax of said given 
compound media document, and is obtained as a result of being analyzed, Accumulate said 
compound media document and said extracted characteristic information, and a compound 
media document is inputted as a search condition, The 2nd characteristic information is 
extracted from a component of said search condition which analyzes the syntax of said 
inputted search condition, and is acquired as a result of being analyzed, A similar retrieval 
method of a compound media document which judges similarity of two compound media 
documents based on said 1st characteristic information accumulated and said 2nd 
characteristic information, and is characterized by outputting a similar compound media 
document. 

[Claim 2]A similar retrieval method of the compound media document according to claim 1 
which inputs a document which a user illustrated as a search key at the time of search, 
extracts said 2nd characteristic information from an illustrated document, and calculates 
similarity between documents by said 2nd extracted characteristic information and said 1st 
characteristic information. 

[Claim 3]A similar retrieval method of a compound media document given in claims 1 and 2 
which set up an evaluation value based on a similarity decision result for every component with 
information on media and structure information characterized by comprising the following as 
whole compound media document similarity. 

Information and structure information on media containing a text, a picture, and a sound which 
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constitute said document illustrated as said search key when calculating similarity of said 
compound media document. 

A text, a picture, a sound which constitute said document accumulated. 

[Claim 4]As a similarity decision result for said every component, text information of said 
illustrated document, A similarity decision result of text information of said accumulated 
document, and said illustrated picture information of a document, A similarity decision result 
with picture information of an accumulated this document, and speech information of a this 
illustrated document, A similar retrieval method of the compound media document according to 
claim 3 using a similarity decision result of a similarity decision result with speech information 
of an accumulated this document, structure information on a this illustrated document, and 
structure information on a this accumulated document. 

[Claim 5]Text information included in said document when calculating similarity of said 
compound media document, A similar retrieval method of the compound media document 
according to claim 3 which makes what calculated similarity, applied a value of dignity to said 
similarity for every characteristic information of picture information, speech information, and 
structure information, and took linear combination similarity as said whole compound media 
document. 

[Claim 6]When calculating similarity of said compound media document and two or more same 
media exist in the same document, About all the search keys for every media including text 
information, picture information, and speech information which are included in said illustrated 
document. A similar retrieval method of a compound media document given in claims 2 and 3 
which calculate all the similarity to this search key in these media in an accumulated document 
and to which said similarity makes the highest thing similarity of representation to said search 
key. 

[Claim 7]When calculating similarity of said compound media document, as said search key, 
Said tree with an order label which expressed each of structure information on said illustrated 
document, and structure information on said accumulated document as a tree with an order 
label (ordered labeled tree), and expressed said illustrated document, A similar retrieval 
method of the compound media document according to claim 1 of setting up similarity of 
structure information on a document by comparing shape with a tree with an order label 
expressing said accumulated document. 

[Claim 8]When setting up similarity of said structure information, it is considered that said 
document structure is a tree, The number of times which performed editing operation including 
insertion of a node required in order to change into a tree showing said document accumulated 
from a tree showing said illustrated document, deletion of a node, and change of a node name, 
A similar retrieval method of the compound media document according to claim 7 of setting up 
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compilation distance computed from cost required to perform this editing operation as similarity 
of said document. 

[Claim 9]Text information included in a document when calculating similarity of a compound 
media document, Similarity based on characteristic information of picture information, speech 
information, and structure information is calculated, A similar retrieval method of the compound 
media document according to claim 3 which makes similarity based on characteristic 
information of text information and picture information which perform the 1st-step selection 
based on a similarity calculation result based on characteristic information of structure 
information, and are contained in said document, or speech information similarity as the whole 
compound media document. 

[Claim 10]Text information included in a document when calculating similarity of a compound 
media document, Similarity based on characteristic information of picture information, speech 
information, and structure information is calculated, A similar retrieval method of the compound 
media document according to claim 3 which makes similarity based on characteristic 
information of structure information which performs the 1st-step selection based on a similarity 
calculation result based on characteristic information of said text information, said picture 
information, or said speech information, and is included in said document similarity as the 
whole compound media document. 

[Claim 1 1]A similar retrieval method of a compound media document given in claims 2 and 3 
which set up similarity in a document level of these media when calculating similarity of a 
compound media document and two or more same media exist in a document illustrated as a 
search key. 

[Claim 12]When setting up similarity in a document level of said media, about each search key 
of two or more said media in a document illustrated as a search key. About all the search keys 
for every media including text information, picture information, and speech information which 
are included in said illustrated document. All the similarity to this search key in these media in 
an accumulated document is calculated, A similar retrieval method of the compound media 
document according to claim 1 1 that said similarity makes the highest thing similarity of 
representation to said search key, calculates average value of similarity of said representation, 
and sets up similarity in a document level of said media. 

[Claim 13]When setting up similarity in a document level of said media, about each search key 
of two or more said media in a document illustrated as a search key. About all the search keys 
for every media including text information, picture information, and speech information which 
are included in said illustrated document. All the similarity to this search key in these media in 
an accumulated document is calculated, A similar retrieval method of the compound media 
document according to claim 1 1 that said similarity makes the highest thing similarity of 
representation to said search key, and makes what has the highest similarity similarity in a 
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document level of said media among similarity of said representation. 
[Claim 14]When performing similar retrieval of a compound media document, each of structure 
information on a document illustrated as a search key and structure information on an 
accumulated document is expressed as a tree with an order label (ordered labeled tree), 
Characteristic information of each media in each document is expressed as a tree with an 
order label with an attribute stored as an attribute of a node in said tree with an order label, A 
similar retrieval method of the compound media document according to claim 7 of setting up 
similarity of a compound media document by comparing an attribute and shape with a tree with 
an order label with an attribute expressing a tree with an order label with an attribute 
expressing said illustrated document, and said accumulated document. 
[Claim 15]When performing similar retrieval of a compound media document, about a tree with 
an order label with an attribute expressing an accumulated document with a node which has 
characteristic information which is the attribute of each node of a tree with an order label with 
an attribute expressing said illustrated document, and similar characteristic information as an 
attribute. A similar retrieval method of the compound media document according to claim 14 of 
setting up similarity of a compound media document from a difference in structural physical 
relationship of said node. 

[Claim 16]When setting up similarity of said structure information, it is considered that 
document structure is a tree with an order label, A similar retrieval method of the compound 
media document according to claim 7 of evaluating and mapping this characteristic information 
and setting up distance on said vector space as similarity of a document on multi dimensional 
vector space based on characteristic information about this tree with an order label. 
[Claim 17]A similar retrieval method of the compound media document according to claim 16 
which calculates similarity of a document by evaluating and using position information on a 
name and a node number of each node of said tree with an order label, or each node as 
characteristic information about said label tree with an order. 

[Claim 18]lf it is the frequency of occurrence of a concept which a descriptive content of a text 
expresses as said characteristic information if it is text information, or each word, and picture 
information, If it is hue of a picture, chroma saturation, luminosity, color arrangement, and 
speech information and is strength of a sound, a melody, and structure information, A similar 
retrieval method of the compound media document according to claim 1 which makes shape of 
a tree when a tree with an order label expresses document structure, a label name of a node, 
link information, etc. characteristic information from which it is extracted from a component of 
said compound media document. 

[Claim 19]An evaluation value based on a similarity decision result for every component with 
information on media and structure information characterized by comprising the following, A 
similar retrieval method of the compound media document according to claim 1 of setting up as 
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whole compound media document similarity, carrying out ranking by arranging similarity of said 

accumulated document in a descending order, and judging similarity. 

Information and structure information on media containing a text, a picture, and a sound which 

constitute said document illustrated considering similarity of an accumulated document to said 

illustrated document as said search key when judging said similarity. 

A text, a picture, a sound which constitute said document accumulated. 

[Claim 20]When setting up said similarity, for every component of said compound media 
document. Text information, picture information and speech information which set up similarity 
and are included in said document, And a similar retrieval method of the compound media 
document according to claim 3 which makes what calculated similarity, applied a value of 
dignity to said similarity for every characteristic information of structure information, and took 
linear combination similarity as said whole compound media document. 
[Claim 21]When setting up similarity of said whole compound media document, as an 
evaluation value based on a similarity decision result for every component of said document 
The similarity of each component itself. Or a similar retrieval method of the compound media 
document according to claim 3 of using what multiplied dignity given to similarity of each 
component by user. 

[Claim 22]A similarity retrieval device of a compound media document which is a structured 
document which adds to text information, picture information, and voice data that is data of 
human being's voice characterized by comprising the following, and comprises speech 
information containing composition data containing CD and a record and music data. 
A compound media document input means which inputs a compound media document. 
Said compound media document given by said compound media document input means, and 
a document analyzing means which analyzes the syntax of an inputted search condition. 
A characteristic information extraction means to extract characteristic information from a 
component of a document obtained as a result of being analyzed by said document analyzing 
means. 

An accumulation means which accumulates said characteristic information extracted by said 
compound media document and said characteristic information extraction means, A search 
condition input means which inputs a compound media document as a search condition, and 
characteristic information of said compound media document accumulated in said 
accumulation means, A document comparison means to judge similarity of two compound 
media documents based on characteristic information extracted by said characteristic 
information extraction means based on a result of having analyzed said inputted search 
condition by said document analyzing means, An output means which outputs a similar 
compound media document based on similarity judged by said document comparison means. 
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[Claim 23]A similarity retrieval device of the compound media document according to claim 22 
characterized by comprising the following. 

An input sentence document characteristic information extraction means to extract a 
compound media document illustrated as a search key including a means to input a document 
in which a user illustrated said search condition input means as a search key from said 
compound media document in which said characteristic information extraction means was 
given. 

Input sentence document characteristic information by which said document comparison 
means was extracted from a document illustrated by said user by said input sentence 
document characteristic information extraction means including a search characteristic 
information extraction means to extract characteristic information of said search information. 
A similarity calculation means to calculate similarity between compound media documents 
illustrated as said compound media document and said search key by search characteristic 
information extracted by said search characteristic information extraction means. 

[Claim 24]A similarity retrieval device of a compound media document given in claims 21 and 
22 characterized by comprising the following. 

Text information, picture information, speech information, and structure information that said 
similarity calculation means constitutes said illustrated document from said search condition 
input means as said search key. 

A similarity setting-out means to set up an evaluation value based on a similarity decision 
result for every component with text information, picture information, speech information, and 
structure information which constitute said compound media document accumulated in said 
accumulation means as similarity of the whole compound media document. 

[Claim 25]A similarity decision result of text information of said document in which said 
similarity setting-out means is accumulated in text information and said accumulation means of 
said illustrated document as a similarity decision result for said every component, A similarity 
decision result of picture information of an illustrated this document, and picture information of 
a document accumulated in this accumulation means, A similarity retrieval device of the 
compound media document according to claim 24 using a similarity decision result of speech 
information of an illustrated this document, and speech information of a document 
accumulated in this accumulation means, and a similarity decision result of structure 
information on this illustration **** document, and structure information on a document 
accumulated in this accumulation means. 

[Claim 26]Text information, picture information and speech information by which a front 
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similarity setting-out means is contained in said document, And a similarity retrieval device of 
the compound media document according to claim 24 including a linear combination 
calculating means which makes what calculated similarity, applied a value of dignity to this 
similarity for every characteristic information of structure information, and took linear 
combination similarity as said whole compound media document. 
[Claim 27]When two or more same media exist in the same document, said document 
comparison means, About all the search keys for every media containing a text, a picture, and 
a sound which are contained in said illustrated document. A similarity retrieval device of a 
compound media document given in claims 22 and 23 containing a representation similarity 
determination means by which all the similarity to this search key in media in an accumulated 
document is calculated, and this similarity makes the highest thing similarity of representation 
to said search key. 

[Claim 28]A similarity retrieval device of the compound media document according to claim 23 
characterized by comprising the following. 

Said tree with an order label which said similarity calculation means expressed each of 
structure information on said illustrated document, and structure information on an 
accumulated document as a tree with an order label (ordered labeled tree) as said search key, 
and expressed said illustrated document. 

A tree shape comparison means with an order label to set up similarity of structure information 
on a document by comparing shape with a tree with an order label expressing said 
accumulated document. 

[Claim 29]A similarity retrieval device of the compound media document according to claim 28 
characterized by comprising the following. 

The number of times which performed editing operation including insertion of a node required 
in order to change said tree shape comparison means with an order label into a tree showing 
said document accumulated from a tree which considers that said document structure is a tree 
and expresses said illustrated document when judging similarity of said structure information, 
deletion of a node, and change of a node name. 

A compilation distance calculating means which sets up compilation distance computed from 
cost required to perform this editing operation as similarity of said document. 

[Claim 30]A similarity retrieval device of the compound media document according to claim 24 
characterized by comprising the following. 

A means to calculate similarity based on characteristic information of text information, picture 
information and speech information, and structure information that said similarity calculation 
means is contained in a document. 
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A means to perform the 1st-step selection based on a similarity calculation result based on 
characteristic information of structure information. 

A means which makes similarity based on characteristic information of text information and 
picture information which are contained in said document, or speech information similarity as 
the whole compound media document. 

[Claim 31]A similarity retrieval device of the compound media document according to claim 24 
characterized by comprising the following. 

A means to calculate similarity based on characteristic information of text information, picture 
information and speech information, and structure information that said similarity calculation 
means is contained in a document. 

A means to perform the 1st-step selection based on a similarity calculation result based on 
characteristic information of said text information, said picture information, or said speech 
information. 

A means which makes similarity based on characteristic information of structure information 
included in said document similarity as the whole compound media document. 

[Claim 32]A similarity retrieval device of a compound media document given in claims 23 and 
24 which contain a document level similarity calculation means to set up similarity in a > 
document level of these media when two or more same media exist in a document in which 
said similarity calculation means was illustrated as a search key. 

[Claim 33]A similarity retrieval device of the compound media document according to claim 32 
characterized by comprising the following. 

Said document level similarity calculation means about each search key of two or more said 
media in a document illustrated as a search key. A means to calculate all the similarity to this 
search key in these media in an accumulated document about all the search keys for every 
media including text information, picture information, and speech information which are 
included in said illustrated document. 

A means by which said similarity makes the highest thing similarity of representation to said 
search key. 

A means to calculate average value of similarity of said representation. 

A document level similarity setting-out means to set up similarity in a document level of said 

media. 

[Claim 34]A similarity retrieval device of the compound media document according to claim 32 
characterized by comprising the following. 

Said document level similarity setting-out means about each search key of two or more said 
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media in a document illustrated as a search key. A means to calculate all the similarity to this 
search key in these media in an accumulated document about all the search keys for every 
media including text information, picture information, and speech information which are 
included in an illustrated this document. 

A means by which said similarity makes the highest thing similarity of representation to said 
search key. 

A means which makes what has the highest similarity similarity in a document level of said 
media among similarity of said representation. 

[Claim 35]Said similarity calculation means expresses each of structure information on a 
document illustrated as a search key, and structure information on an accumulated document 
as a tree with an order label (ordered labeled tree), Characteristic information of each media in 
each document is expressed as a tree with an order label with an attribute stored as an 
attribute of a node in said tree with an order label, A similarity retrieval device of the compound 
media document according to claim 28 which contains a similar retrieval means to set up 
similarity of a compound statement document, by comparing an attribute and shape with a tree 
with an order label with an attribute expressing a tree with an order label with an attribute 
expressing said illustrated document, and said accumulated document. 
[Claim 36]Said similar retrieval means about a tree with an order label with an attribute 
expressing an accumulated document with a node which has characteristic information which 
is the attribute of each node of a tree with an order label with an attribute expressing an 
illustrated document, and similar characteristic information as an attribute. A similarity retrieval 
device of the compound media document according to claim 35 containing a means to set up 
similarity of a compound media document from a difference in structural physical relationship 
of this node. 

[Claim 37]A similarity retrieval device of the compound media document according to claim 30 
characterized by comprising the following. 

A means for said tree shape comparison means with an order label to consider that document 
structure is a tree with an order label, and to evaluate and map this characteristic information 
on multi dimensional vector space based on characteristic information about this tree with an 
order label. 

A means to set up distance on said vector space as similarity of a document. 

[Claim 38]A similarity retrieval device of the compound media document according to claim 37 
which calculates similarity of a document by evaluating and using position information on a 
name and a node number of each node of said tree with an order label, or each node as 
characteristic information about said label tree with an order. 
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[Claim 39]lf it is the frequency of occurrence of a concept which a descriptive content of a text 
expresses as said characteristic information if it is text information, or each word, and picture 
information, If it is hue of a picture, chroma saturation, luminosity, color arrangement, and 
speech information and is strength of a sound, a melody, and structure information, A similarity 
retrieval device of the compound media document according to claim 22 which makes shape 
of a tree when a tree with an order label expresses document structure, a label name of a 
node, link information, etc. characteristic information from which it is extracted from a 
component of said compound media document. 

[Claim 40]A similarity retrieval device of the compound media document according to claim 22 
characterized by comprising the following. 

Information and structure information on media containing a text, a picture, and a sound which 
constitute said document in which said document comparison means was illustrated 
considering similarity of an accumulated document to said illustrated document as said search 
key. 

A means to set up an evaluation value based on a similarity decision result for every 
component with information on media and structure information containing a text, a picture, 
and a sound which constitute said document accumulated as whole compound media 
document similarity. 

A means to carry out ranking by arranging similarity of said accumulated document in a 
descending order, and to judge similarity. 

[Claim 41]A similarity retrieval device of the compound media document according to claim 24 
characterized by comprising the following. 

A means by which said similarity setting-out means sets up similarity for every component of 
said compound media document. 

A means to calculate similarity for every characteristic information of text information included 
in said document, picture information and speech information, and structure information. 
A means which makes what applied a value of dignity to said similarity and took linear 
combination similarity as said whole compound media document. 

[Claim 42]A similarity retrieval device of the compound media document according to claim 24 
in which said similarity setting-out means contains a means to use the similarity of each 
component itself, or a thing which multiplied dignity given to similarity of each component by 
user, as an evaluation value based on a similarity decision result for every component of said 
document. 

[Claim 43]A storage which added to text information, picture information, and voice data that is 
data of human being's voice characterized by comprising the following, and stored a similar 
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retrieval program of a compound media document which is a structured document which 
comprises speech information containing composition data containing CD and a record and 
music data. 

Said given compound media document and a document analyzing process of analyzing the 
syntax of an inputted search condition. 

A characteristic information extraction process of extracting characteristic information from a 
component of a document obtained as a result of being analyzed in said document analyzing 
process. 

A storing process of storing in a memory measure said characteristic information extracted in 
said compound media document and said characteristic information extraction process. 
A search condition input process of making a compound media document inputting as a 
search condition, Characteristic information of said compound media document accumulated 
in said memory measure, A document comparison process of judging similarity of two 
compound media documents from characteristic information extracted in said characteristic 
information extraction process based on a result of having analyzed said inputted search 
condition in said document analyzing process, An output process to which a similar compound 
media document is made to output based on similarity judged in said document comparison 
process. 

[Claim 44]A storage which stored a similar retrieval program of the compound media document 
according to claim 43, comprising: 

An input sentence document characteristic information extraction process of extracting 
characteristic information from said compound media document in which said characteristic 
information extraction process was given including a process of inputting a document in which 
a user illustrated said search condition input process as a search key. 
Input sentence document characteristic information from which said document comparison 
process was extracted from a document illustrated by said user in said input sentence 
document characteristic information extraction process including a search characteristic 
information extraction process of extracting characteristic information of a compound media 
document illustrated as a search key. 

A similarity calculation process of calculating similarity between compound media documents 
illustrated as said compound media document and said search key by search characteristic 
information extracted in said search characteristic information extraction process. 

[Claim 45]A storage which stored a similar retrieval program of a compound media document 
given in claims 43 and 44, comprising: 

Text information, picture information, speech information, and structure information that said 



http://www4.ipdl.inpit.go.jp^ 12/24/2008 



JP,2000-148793,A [CLAIMS] 



Page 12 of 16 



similarity calculation process constitutes a document illustrated as said search key from said 
search condition input process. 

A similarity setting process which sets up an evaluation value based on a similarity decision 
result for every component with text information, picture information, speech information, and 
structure information which constitute said compound media document accumulated in said 
memory measure as similarity of the whole compound media document. 

[Claim 46]Said similarity setting process as a similarity decision result for said every 
component, A similarity decision result of text information of said illustrated document, and text 
information of said document accumulated in said memory measure, A similarity decision 
result of picture information of an illustrated this document, and picture information of a 
document accumulated in this memory measure, A similarity decision result of speech 
information of an illustrated this document, and speech information of a document 
accumulated in this memory measure, A storage which stored a similar retrieval program of the 
compound media document according to claim 45 using a similarity decision result of structure 
information on this illustration **** document, and structure information on a document 
accumulated in this memory measure. 

[Claim 47]Text information, picture information and speech information by which a front 
similarity setting process is included in said document, And a storage which stored a similar 
retrieval program of the compound media document according to claim 45 including a linear 
combination calculation process of making into similarity as said whole compound media 
document what calculated similarity, applied a value of dignity to this similarity for every 
characteristic information of structure information, and took linear combination. 
[Claim 48]When two or more same media exist in the same document, said document 
comparison process, About all the search keys for every media containing a text, a picture, 
and a sound which are contained in said illustrated document. A storage which stored a similar 
retrieval program of a compound media document given in claims 43 and 44 including a 
representation similarity determination process that calculate all the similarity to this search 
key in media in an accumulated document, and this similarity makes the highest thing similarity 
of representation to said search key. 

[Claim 49]A storage which stored a similar retrieval program of the compound media document 
according to claim 44, comprising: 

Said tree with an order label which said similarity calculation process expressed each of 
structure information on said illustrated document, and structure information on an 
accumulated document as a tree with an order label (ordered labeledtree) as said search key, 
and expressed said illustrated document. 

A tree shape comparison process with an order label of setting up similarity of structure 
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information on a document by comparing shape with a tree with an order label expressing said 
accumulated document. 

[Claim 50]A storage which stored a similar retrieval program of the compound media document 
according to claim 49, comprising: 

The number of times which performed editing operation including insertion of a node required 
in order to change said tree shape comparison process with an order label into a tree showing 
said document accumulated from a tree which considers that said document structure is a tree 
and expresses said illustrated document when judging similarity of said structure information, 
deletion of a node, and change of a node name. 

A compilation distance calculation process of setting up compilation distance computed from 
cost required to perform this editing operation as similarity of said document. 

[Claim 51]A storage which stored a similar retrieval program of the compound media document 
according to claim 44, comprising: 

A process of calculating similarity based on characteristic information of text information, 
picture information and speech information, and structure information that said similarity 
calculation process is included in a document. 

A process of performing the 1st-step selection based on a similarity calculation result based on 
characteristic information of structure information. 

A process of making into similarity as the whole compound media document similarity based 
on characteristic information of text information and picture information which are contained in 
said document, or speech information. 

[Claim 52]Claim 43 and a storage which stored a similar retrieval program of a compound 
media document given in 44 characterized by comprising the following. 
A process of calculating similarity based on characteristic information of text information, 
picture information and speech information, and structure information that said similarity 
calculation process is included in a document. 

A process of performing the 1st-step selection based on a similarity calculation result based on 
characteristic information of said text information, said picture information, or said speech 
information. 

A process of making into similarity as the whole compound media document similarity based 
on characteristic information of structure information included in said document. 

[Claim 53]A storage which stored a similar retrieval program of a compound media document 
given in claims 43 and 44 which include a document level similarity calculation process of 
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setting up similarity in a document level of these media when two or more same media exist in 
a document in which said similarity calculation process was illustrated as a search key. 
[Claim 54]A storage which stored a similar retrieval program of the compound media document 
according to claim 53, comprising: 

Said document level similarity calculation process about each search key of two or more said 
media in a document illustrated as a search key. A process of calculating all the similarity to 
this search key in these media in an accumulated document about all the search keys for 
every media including text information, picture information, and speech information which are 
included in said illustrated document. 

A process of making a thing with said highest similarity into similarity of representation to said 
search key. 

A process of calculating average value of similarity of said representation. 

A document level similarity setting process which sets up similarity in a document level of said 

media. 

[Claim 55]A storage which stored a similar retrieval program of the compound media document 
according to claim 54, comprising: 

Said document level similarity setting process about each search key of two or more said 
media in a document illustrated as a search key. A process of calculating all the similarity to 
this search key in these media in an accumulated document about all the search keys for 
every media including text information, picture information, and speech information which are 
included in an illustrated this document. 

A process of making a thing with said highest similarity into similarity of representation to said 
search key. 

A process of making into similarity in a document level of said media what has the highest 
similarity among similarity of said representation. 

[Claim 56]Said similarity calculation process expresses each of structure information on a 
document illustrated as a search key, and structure information on an accumulated document 
as a tree with an order label (ordered labeled tree), Characteristic information of each media in 
each document is expressed as a tree with an order label with an attribute stored as an 
attribute of a node in said tree with an order label, By comparing an attribute and shape with a 
tree with an order label with an attribute expressing a tree with an order label with an attribute 
expressing said illustrated document, and said accumulated document. A storage which stored 
a similar retrieval program of the compound media document according to claim 49 including a 
similar retrieval process of setting up similarity of a compound statement document. 
[Claim 57]Said similar retrieval process about a tree with an order label with an attribute 
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expressing an accumulated document with a node which has characteristic information which 
is the attribute of each node of a tree with an order label with an attribute expressing an 
illustrated document, and similar characteristic information as an attribute. A storage which 
stored a similar retrieval program of the compound media document according to claim 56 
including a process of setting up similarity of a compound media document from a difference in 
structural physical relationship of this node. 

[Claim 58]A storage which stored a similar retrieval program of the compound media document 
according to claim 49, comprising: 

A process of said tree shape comparison process with an order label considering that 
document structure is a tree with an order label, and evaluating and mapping this characteristic 
information on multi dimensional vector space based on characteristic information about this 
tree with an order label. 

A process of setting up distance on said vector space as similarity of a document. 

[Claim 59]A storage which stored a similar retrieval program of the compound media document 
according to claim 58 which calculates similarity of a document by evaluating and using 
position information on a name and a node number of each node of said tree with an order 
label, or each node as characteristic information about said label tree with an order. 
[Claim 60]lf it is the frequency of occurrence of a concept which a descriptive content of a text 
expresses as said characteristic information if it is text information, or each word, and picture 
information, If it is hue of a picture, chroma saturation, luminosity, color arrangement, and 
speech information and is strength of a sound, a melody, and structure information, A storage 
which stored a similar retrieval program of the compound media document according to claim 
43 made into characteristic information from which shape of a tree when a tree with an order 
label expresses document structure, a label name of a node, link information, etc. are 
extracted from a component of said compound media document. 

[Claim 61]A storage which stored a similar retrieval program of the compound media document 
according to claim 43, comprising: 

Information and structure information on media containing a text, a picture, and a sound which 
constitute said document in which said document comparison process was illustrated 
considering similarity of an accumulated document to said illustrated document as said search 
key. 

A process of setting up an evaluation value based on a similarity decision result for every 
component with information on media and structure information containing a text, a picture, 
and a sound which constitute said document accumulated as whole compound media 
document similarity. 

A process of carrying out ranking by arranging similarity of said accumulated document in a 
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descending order, and judging similarity. 

[Claim 62]A storage which stored a similar retrieval program of the compound media document 
according to claim 44, comprising: 

A process to which said similarity setting process sets similarity for every component of said 
compound media document. 

A process of calculating similarity for every characteristic information of text information 
included in said document, picture information and speech information, and structure 
information. 

A process of making into similarity as said whole compound media document what applied a 
value of dignity to said similarity and took linear combination. 

[Claim 63]Said similarity setting process as an evaluation value based on a similarity decision 
result for every component of said document The similarity of each component itself. Or a 
storage which stored a similar retrieval program of the compound media document according 
to claim 44 including a process of using what multiplied dignity given to similarity of each 
component by user. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the 
original precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the lnvention]This invention relates to the storage which stored the similar retrieval 
program of the similar retrieval method of a compound media document, the device, and the 
compound media document, It is related with the storage which stored the similar retrieval 
program of the similar retrieval method of the compound media document for searching a 
similar compound media document about a compound media document data base especially, 
the device, and the compound media document. 
[0002] 

[Description of the Prior Art]As a Prior art about a structured document search method, there 
are combination (Kanamoto, Kato, Kinutani, Yoshikawa, the "structured document searching 
method in which efficient updating is possible") of Structurelndex of the Nara Institute of 
Science and Technology study and Content Index, etc. In these systems, indexes, such as an 
inverted file about the appearing position of the text information stored in structured 
documents, such as SGML documents and an XML document, or the information on document 
structure, including element name etc., are prepared beforehand, The combination of text 
information, or text information and structure information is given as a search key, Search of a 
structured document is enabled by performing the Boolean retrieval which judges whether the 
given keyword is contained in the document, and range retrieval which judges whether the 
keyword given within specified limits appears. 

[0003]By systems, such as ConceptBase of JUST System, and VextSearch of Komatsu Soft. 
The concept included in the text used as the concept included in the text inputted by the 
natural sentence etc. as a search key and a retrieval object is compared, and search of a 
similar document (only text) is enabled. 
[0004] 
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[Problem(s) to be Solved by the lnvention]However, since the similar retrieval method for a 
compound media document is not established when it is going to apply the above-mentioned 
conventional method to the similar retrieval for a compound media document, the following 
problems occur. 

- If a user does not know beforehand the information about document structures, such as an 
element name (notes: tag name showing structure information) in a structured document, 
search which gave structure information to a part of search key cannot be performed. 
[0005]- Similar retrieval of the compound media document which uses the information and 
structure information on media other than texts, such as a picture and a sound, as a search 
key cannot be performed. 

If it was made in view of the above-mentioned point and a compound media document is 
illustrated as a search condition, this invention, Characteristic information included in the 
illustrated document, such as text information, a picture and audio information, and structure 
information, and the text information included in the accumulated document, Compare 
characteristic information, such as a picture, audio information, and structure information, 
respectively, and The similarity of text information, By calculating the similarity of picture 
information, the similarity of speech information, and the similarity of structure information 
separately, multiplying the value of dignity by them, making into the similarity in a document 
level what calculated the synthetic evaluation value, and adjusting the value of dignity. It aims 
at providing the storage which stored the similar retrieval program of the similar retrieval 
method of the compound media document in which the similar retrieval which thought the 
similarity of text information as important, the similar retrieval which thought the similarity of 
picture information as important, the similar retrieval which thought the similarity of structure 
information as important, etc. are possible, the device, and the compound media document. 
[0006] 

[Means for Solving the Problem] Drawing 1 is a figure for explaining a principle of this invention. 
This invention (claim 1) is added to text information, picture information, and voice data that is 
data of human being's voice, In a similar retrieval method of a compound media document 
which is a structured document which comprises speech information containing composition 
data containing CD and a record and music data, The 1st characteristic information is 
extracted from a component of a compound media document which analyzes the syntax of a 
given compound media document (Step 1), and is obtained as a result of being analyzed (Step 
2), Accumulate a document and extracted characteristic information (Step 3), and a compound 
media document is inputted as a search condition (Step 4), The 2nd characteristic information 
is extracted from a component of a search condition which analyzes the syntax of an inputted 
search condition (Step 5), and is acquired as a result of being analyzed (Step 6), Similarity of 
two compound media documents is judged based on the 1st characteristic information and 2nd 
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characteristic information that are accumulated (Step 7), and a similar compound media 
document is outputted (Step 8). 

[0007]At the time of search, this invention (claim 2) inputs a document which a user illustrated 
as a search key, extracts the 2nd characteristic information from an illustrated document, and 
calculates similarity between documents by the 2nd characteristic information and 1st 
characteristic information that were extracted. Information and structure information on media 
containing a text, a picture, and a sound which constitute a document illustrated as a search 
key when this invention (claim 3) calculates similarity of a compound media document, An 
evaluation value based on a similarity decision result for every component with information on 
media and structure information containing a text, a picture, and a sound which constitute a 
document accumulated is set up as similarity of the whole compound media document. 
[0008]Text information of a document in which this invention (claim 4) was illustrated as a 
similarity decision result for every component, A similarity decision result of text information of 
an accumulated document, and picture information of an illustrated document, A similarity 
decision result of a similarity decision result of a similarity decision result with picture 
information of an accumulated this document, speech information of a this illustrated 
document, and speech information of a this accumulated document, structure information on a 
this illustrated document, and structure information on a this accumulated document is used. 
[0009]When this invention (claim 5) calculates similarity of a compound media document, for 
every characteristic information of text information included in a document, picture information 
and speech information, and structure information, it calculates similarity, applies a value of 
dignity to similarity, and makes what took linear combination similarity as the whole compound 
media document. 

[0010]When this invention (claim 6) calculates similarity of a compound media document, Text 
information included in an illustrated document when two or more same media exist in the 
same document, All the similarity to this search key in these media in an accumulated 
document is calculated about all the search keys for every media including picture information 
and speech information, and similarity makes the highest thing similarity of representation to a 
search key. 

[001 1]When this invention (claim 7) calculates similarity of a compound media document, as a 
search key, A tree with an order label which expressed each of structure information on an 
illustrated document, and structure information on an accumulated document as a tree with an 
order label (ordered labeled tree), and expressed an illustrated document, Similarity of 
structure information on a document is set up by comparing shape with a tree with an order 
label expressing an accumulated document. 

[0012]When this invention (claim 8) sets up similarity of structure information, it considers that 
document structure is a tree, The number of times which performed editing operation including 
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insertion of a node required in order to change into a tree showing a document accumulated 
from a tree showing an illustrated document, deletion of a node, and change of a node name, 
and compilation distance computed from cost required to perform this editing operation are set 
up as similarity of a document. 

[0013]When this invention (claim 9) calculates similarity of a compound media document, 
Similarity based on characteristic information of text information included in a document, 
picture information and speech information, and structure information is calculated, The 1st- 
step selection based on a similarity calculation result based on characteristic information of 
structure information is performed, and let similarity based on characteristic information of text 
information and picture information which are contained in a document, or speech information 
be the similarity as the whole compound media document. 

[0014]When this invention (claim 10) calculates similarity of a compound media document, 
Similarity based on characteristic information of text information included in a document, 
picture information and speech information, and structure information is calculated, The 1st- 
step selection based on a similarity calculation result based on characteristic information of 
text information, picture information, or speech information is performed, and let similarity 
based on characteristic information of structure information included in a document be the 
similarity as the whole compound media document. 

[0015]This invention (claim 11) sets up similarity in a document level of these media, when 
calculating similarity of a compound media document and two or more same media exist in a 
document illustrated as a search key. When this invention (claim 12) sets up similarity in a 
document level of media, About each search key of two or more media in a document 
illustrated as a search key. About all the search keys for every media including text 
information, picture information, and speech information which are included in an illustrated 
document. All the similarity to this search key in these media in an accumulated document is 
calculated, and similarity makes the highest thing similarity of representation to a search key, 
calculates average value of similarity of representation, and sets up similarity in a document 
level of media. 

[0016]When this invention (claim 13) sets up similarity in a document level of media, About 
each search key of two or more media in a document illustrated as a search key. About all the 
search keys for every media including text information, picture information, and speech 
information which are included in an illustrated document. All the similarity to this search key in 
these media in an accumulated document is calculated, and similarity makes the highest thing 
similarity of representation to a search key, and makes what has the highest similarity similarity 
in a document level of media among similarity of representation. 
[0017]When this invention (claim 14) performs similar retrieval of a compound media 
document, Each of structure information on a document illustrated as a search key and 
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structure information on an accumulated document is expressed as a tree with an order label 
(ordered labeled tree), Characteristic information of each media in each document is 
expressed as a tree with an order label with an attribute stored as an attribute of a node in a 
tree with an order label, Similarity of a compound media document is set up by comparing an 
attribute and shape with a tree with an order label with an attribute expressing a tree with an 
order label with an attribute expressing an illustrated document, and an accumulated 
document 

[0018]When this invention (claim 15) performs similar retrieval of a compound media 
document, About a tree with an order label with an attribute expressing an accumulated 
document with a node which has characteristic information which is the attribute of each node 
of a tree with an order label with an attribute expressing an illustrated document, and similar 
characteristic information as an attribute. Similarity of a compound media document is set up 
from a difference in structural physical relationship of a node. 

[0019]When it sets up similarity of structure information, this invention (claim 16) considers that 
document structure is a tree with an order label, based on characteristic information about this 
tree with an order label, on multi dimensional vector space, evaluates and maps this 
characteristic information and sets up distance on vector space as similarity of a document. 
This invention (claim 17) calculates similarity of a document by evaluating and using position 
information on a name and a node number of each node of a tree with an order label, or each 
node as characteristic information about a label tree with an order. 
[0020]As characteristic information, if this invention (claim 18) is text information, If it is the 
frequency of occurrence of a concept which a descriptive content of a text expresses, or each 
word, and picture information, If it is hue of a picture, chroma saturation, luminosity, color 
arrangement, and speech information and is strength of a sound, a melody, and structure 
information, let shape of a tree when a tree with an order label expresses document structure, 
a label name of a node, link information, etc. be the characteristic information from which it is 
extracted from a component of a compound media document. 

[0021 Information and structure information on media containing a text, a picture, and a sound 
which constitute a document illustrated considering similarity of an accumulated document to 
an illustrated document as a search key when this invention (claim 19) judged similarity, 
Ranking is carried out by setting up an evaluation value based on a similarity decision result 
for every component with information on media and structure information containing a text, a 
picture, and a sound which constitute a document accumulated as whole compound media 
document similarity, and arranging similarity of an accumulated document in a descending 
order, and similarity is judged. 

[0022]When this invention (claim 20) sets up similarity, for every component of a compound 
media document. Similarity is set up, for every characteristic information of text information 
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included in a document, picture information and speech information, and structure information, 
similarity is calculated, and a value of dignity is applied to similarity, and let what took linear 
combination be the similarity as the whole compound media document. 
[0023]When this invention (claim 21) sets up similarity of the whole compound media 
document, it uses the similarity of each component itself, or a thing which multiplied dignity 
given to similarity of each component by user as an evaluation value based on a similarity 
decision result for every component of a document. Drawing 2 is a principle lineblock diagram 
of this invention. 

[0024]This invention (claim 22) is provided with the following. 

The compound media document input means 10 which is a similarity retrieval device of a 
compound media document which is a structured document which comprises speech 
information which adds to text information, picture information, and voice data that is data of 
human being's voice, and contains composition data containing CD and a record and music 
data, and inputs a compound media document. 

A compound media document given by the compound media document input means 10, and 
the document analyzing means 40 which analyzes the syntax of an inputted search condition. 
A characteristic information extraction means 50 to extract characteristic information from a 
component of a document obtained as a result of being analyzed by the document analyzing 
means 40. 

The accumulation means 60 which accumulates characteristic information extracted by 
compound media document and the characteristic information extraction means 50, The 
search condition input means 30 which inputs a compound media document as a search 
condition, A document comparison means 80 to judge similarity of two compound media 
documents based on characteristic information of a compound media document accumulated 
in the accumulation means 60, and characteristic information extracted by the characteristic 
information extraction means 50 based on a result of having analyzed an inputted search 
condition by the document analyzing means 40, The output means 90 which outputs a similar 
compound media document based on similarity judged by the document comparison means 
80. 

[0025]ln the characteristic information extraction means 50 including a means by which this 
invention (claim 23) inputs a document which a user illustrated as a search key in the search 
condition input means 30, An input sentence document characteristic information extraction 
means to extract a compound media document illustrated as a search key from a given 
compound media document, In the document comparison means 80 including a search 
characteristic information extraction means to extract characteristic information of search 
information from a document illustrated by user, A similarity calculation means to calculate 
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similarity between compound media documents illustrated as a compound media document 
and a search key by input sentence document characteristic information extracted by an input 
sentence document characteristic information extraction means and search characteristic 
information extracted by a search characteristic information extraction means is included. 
[0026]Text information, picture information, speech information, and structure information that 
this invention (claim 24) constitutes a document illustrated as a search key by the search 
condition input means 30 in a similarity calculation means, A similarity setting-out means to set 
up an evaluation value based on a similarity decision result for every component with text 
information, picture information, speech information, and structure information which constitute 
a compound media document accumulated in an accumulation means as similarity of the 
whole compound media document is included. 

[0027]ln a similarity setting-out means, this invention (claim 25) as a similarity decision result 
for every component, A similarity decision result of text information of an illustrated document, 
and text information of a document accumulated in an accumulation means, A similarity 
decision result of picture information of an illustrated this document, and picture information of 
a document accumulated in this accumulation means, A similarity decision result of speech 
information of an illustrated this document and speech information of a document accumulated 
in this accumulation means and a similarity decision result of structure information on this 
illustration **** document and structure information on a document accumulated in this 
accumulation means are used. 

[0028]Text information by which this invention (claim 26) is contained in a document in a 
similarity setting-out means, For every characteristic information of picture information, speech 
information, and structure information, similarity is calculated, a value of dignity is applied to 
this similarity, and a linear combination calculating means which makes what took linear 
combination similarity as the whole compound media document is included. In a document 
comparison means, when two or more same media exist in the same document, this invention 
(claim 27), All the similarity to this search key in media in an accumulated document is 
calculated about all the search keys for every media containing a text, a picture, and a sound 
which are contained in an illustrated document, and this similarity contains a representation 
similarity determination means which makes the highest thing similarity of representation to a 
search key. 

[0029]ln a similarity calculation means, this invention (claim 28) as a search key, A tree with an 
order label which expressed each of structure information on an illustrated document, and 
structure information on an accumulated document as a tree with an order label (ordered 
labeled tree), and expressed an illustrated document, By comparing shape with a tree with an 
order label expressing an accumulated document, a tree shape comparison means with an 
order label to set up similarity of structure information on a document is included. 
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[0030]ln a tree shape comparison means with an order label, when this invention (claim 29) 
judges similarity of structure information, The number of times which performed editing 
operation including insertion of a node required in order to change into a tree showing a 
document accumulated from a tree which considers that document structure is a tree and 
expresses an illustrated document, deletion of a node, and change of a node name, A 
compilation distance calculating means which sets up compilation distance computed from 
cost required to perform this editing operation as similarity of a document is included. 
[0031]This invention (claim 30) is provided with the following. 

A means to calculate similarity based on characteristic information of text information included 
in a document, picture information and speech information, and structure information in a 
similarity calculation means. 

A means to perform the 1st-step selection based on a similarity calculation result based on 
characteristic information of structure information. 

A means which makes similarity based on characteristic information of text information and 
picture information which are contained in a document, or speech information similarity as the 
whole compound media document. 

[0032]This invention (claim 31) is provided with the following. 

A means to calculate similarity based on characteristic information of text information included 
in a document, picture information and speech information, and structure information in a 
similarity calculation means. 

A means to perform the 1st-step selection based on a similarity calculation result based on 
characteristic information of text information, picture information, or speech information. 
A means which makes similarity based on characteristic information of structure information 
included in a document similarity as the whole compound media document. 

[0033]ln a similarity calculation means, this invention (claim 32) contains a document level 
similarity calculation means to set up similarity in a document level of these media, when two 
or more same media exist in a document illustrated as a search key. This invention (claim 33) 
is provided with the following. 

About each search key of two or more media in a document illustrated as a search key in a 
document level similarity calculation means. A means to calculate all the similarity to this 
search key in these media in an accumulated document about all the search keys for every 
media including text information, picture information, and speech information which are 
included in an illustrated document. 

A means by which similarity makes the highest thing similarity of representation to a search 
key. 
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A means to calculate average value of similarity of representation. 

A document level similarity setting-out means to set up similarity in a document level of media. 
[0034]This invention (claim 34) is provided with the following. 

About each search key of two or more media in a document illustrated as a search key in a 
document level similarity setting-out means. A means to calculate all the similarity to this 
search key in these media in an accumulated document about all the search keys for every 
media including text information, picture information, and speech information which are 
included in an illustrated this document. 

A means by which similarity makes the highest thing similarity of representation to a search 
key. 

A means which makes what has the highest similarity similarity in a document level of media 
among similarity of representation. 

[0035]This invention (claim 35) expresses each of structure information on a document 
illustrated as a search key, and structure information on an accumulated document as a tree 
with an order label (ordered labeled tree) in a similarity calculation means, Characteristic 
information of each media in each document is expressed as a tree with an order label with an 
attribute stored as an attribute of a node in a tree with an order label, By comparing an 
attribute and shape with a tree with an order label with an attribute expressing a tree with an 
order label with an attribute expressing an illustrated document, and an accumulated 
document, a similar retrieval means to set up similarity of a compound media document is 
included. 

[0036]This invention (claim 36) about a tree with an order label with an attribute expressing an 
accumulated document with a node which has characteristic information which is the attribute 
of each node of a tree with an order label with an attribute which expressed an illustrated 
document in a similar retrieval means, and similar characteristic information as an attribute. A 
means to set up similarity of a compound media document from a difference in structural 
physical relationship of this node is included. 
[0037]This invention (claim 37) is provided with the following. 

A means to consider that document structure is a tree with an order label, and to evaluate and 
map this characteristic information on multi dimensional vector space in a tree shape 
comparison means with an order label based on characteristic information about this tree with 
an order label. 

A means to set up distance on vector space as similarity of a document. 

[0038]This invention (claim 38) calculates similarity of a document by evaluating and using 

http://ww4.ipdl.inpit.go.jp/cgi-biii/tran_web_cgi_ejje7a 12/24/2008 



JP,2000-148793,A [DETAILED DESCRIPTION] 



Page 10 of 25 



position information on a name and a node number of each node of a tree with an order label, 
or each node as characteristic information about a label tree with an order. As characteristic 
information, if this invention (claim 39) is text information, If it is the frequency of occurrence of 
a concept which a descriptive content of a text expresses, or each word, and picture 
information, If it is hue of a picture, chroma saturation, luminosity, color arrangement, and 
speech information and is strength of a sound, a melody, and structure information, let shape 
of a tree when a tree with an order label expresses document structure, a label name of a 
node, link information, etc. be the characteristic information from which it is extracted from a 
component of a compound media document. 
[0039]This invention (claim 40) is provided with the following. 

Information and structure information on media containing a text, a picture, and a sound which 

constitute a document illustrated considering similarity of an accumulated document to an 

illustrated document as a search key in the document comparison means 80. 

A means to set up an evaluation value based on a similarity decision result for every 

component with information on media and structure information containing a text, a picture, 

and a sound which constitute a document accumulated as whole compound media document 

similarity. 

A means to carry out ranking by arranging similarity of an accumulated document in a 
descending order, and to judge similarity. 

[0040]This invention (claim 41) is provided with the following. 

A means to set up similarity for every component of a compound media document in a 
similarity setting-out means. 

A means to calculate similarity for every characteristic information of text information included 
in a document, picture information and speech information, and structure information. 
A means which makes what applied a value of dignity to similarity and took linear combination 
similarity as the whole compound media document. 

[0041]This invention (claim 42) contains a means to use the similarity of each component itself, 
or a thing which multiplied dignity given to similarity of each component by user as an 
evaluation value based on a similarity decision result for every component of a document, in a 
similarity setting-out means. This invention (claim 43) is provided with the following. 
It adds to text information, picture information, and voice data that is data of human being's 
voice, A compound media document which is the storage which stored a similar retrieval 
program of a compound media document which is a structured document which comprises 
speech information containing composition data containing CD and a record and music data, 
and was given, and a document analyzing process of analyzing the syntax of an inputted 
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search condition. 

A characteristic information extraction process of extracting characteristic information from a 
component of a document obtained as a result of being analyzed in a document analyzing 
process. 

A storing process of storing in a memory measure characteristic information extracted in a 
compound media document and a characteristic information extraction process. 
A search condition input process of making a compound media document inputting as a 
search condition, A document comparison process of judging similarity of two compound 
media documents from characteristic information of a compound media document 
accumulated in a memory measure, and characteristic information extracted in a characteristic 
information extraction process based on a result of having analyzed an inputted search 
condition in a document analyzing process, An output process to which a similar compound 
media document is made to output based on similarity judged in a document comparison 
process. 

[00421ln a characteristic information extraction process including a process as which this 
invention (claim 44) inputs a document which a user illustrated as a search key in a search 
condition input process, An input sentence document characteristic information extraction 
process of extracting characteristic information from a given compound media document, In a 
document comparison process including a search characteristic information extraction process 
of extracting characteristic information of a compound media document illustrated as a search 
key from a document illustrated by user, A similarity calculation process of calculating similarity 
between compound media documents illustrated as a compound media document and a 
search key by input sentence document characteristic information extracted in an input 
sentence document characteristic information extraction process and search characteristic 
information extracted in a search characteristic information extraction process is included. 
[0043]Text information, picture information, speech information, and structure information that 
this invention (claim 45) constitutes a document illustrated as a search key in a search 
condition input process in a similarity calculation process, A similarity setting process which 
sets up an evaluation value based on a similarity decision result for every component with text 
information, picture information, speech information, and structure information which constitute 
a compound media document accumulated in a memory measure as similarity of the whole 
compound media document is included. 

[0044]ln a similarity setting process, this invention (claim 46) as a similarity decision result for 
every component, A similarity decision result of text information of an illustrated document, and 
text information of a document accumulated in a memory measure, A similarity decision result 
of picture information of an illustrated this document, and picture information of a document 
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accumulated in this memory measure, A similarity decision result of speech information of an 
illustrated this document and speech information of a document accumulated in this memory 
measure and a similarity decision result of structure information on this illustration **** 
document and structure information on a document accumulated in this memory measure are 
used. 

[0045]Text information by which this invention (claim 47) is contained in a document in a 
similarity setting process, For every characteristic information of picture information, speech 
information, and structure information, similarity is calculated, a value of dignity is applied to 
this similarity, and a linear combination calculation process of making into similarity as the 
whole compound media document what took linear combination is included. In a document 
comparison process, when two or more same media exist in the same document, this 
invention (claim 48), About all the search keys for every media containing a text, a picture, and 
a sound which are contained in an illustrated document. All the similarity to this search key in 
media in an accumulated document is calculated, and this similarity includes a representation 
similarity determination process of making the highest thing into similarity of representation to 
a search key. 

[0046]ln a similarity calculation process, this invention (claim 49) as a search key, A tree with 
an order label which expressed each of structure information on an illustrated document, and 
structure information on an accumulated document as a tree with an order label (ordered 
labeled tree), and expressed an illustrated document, By comparing shape with a tree with an 
order label expressing an accumulated document, a tree shape comparison process with an 
order label of setting up similarity of structure information on a document is included. 
[0047]ln a tree shape comparison process with an order label this invention (claim 50), The 
number of times which performed editing operation including insertion of a node required 
[ when judging similarity of structure information ] in order to change into a tree showing a 
document accumulated from a tree which considers that document structure is a tree and 
expresses an illustrated document, deletion of a node, and change of a node name, A 
compilation distance calculation process of setting up compilation distance computed from cost 
required to perform this editing operation as similarity of a document is included. 
[0048]This invention (claim 51) is provided with the following. 

A process of calculating similarity based on characteristic information of text information 
included in a document, picture information and speech information, and structure information 
in a similarity calculation process. 

A process of performing the 1st-step selection based on a similarity calculation result based on 
characteristic information of structure information. 

A process of making into similarity as the whole compound media document similarity based 
on characteristic information of text information and picture information which are contained in 
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a document, or speech information. 

[0049]This invention (claim 52) is provided with the following. 

A process of calculating similarity based on characteristic information of text information 
included in a document, picture information and speech information, and structure information 
in a similarity calculation process. 

A process of performing the 1st-step selection based on a similarity calculation result based on 
characteristic information of text information, picture information, or speech information. 
A process of making into similarity as the whole compound media document similarity based 
on characteristic information of structure information included in a document. 

[0050]ln a similarity calculation process, this invention (claim 53) includes a document level 
similarity calculation process of setting up similarity in a document level of these media, when 
two or more same media exist in a document illustrated as a search key. This invention (claim 
54) is provided with the following. 

About each search key of two or more media in a document illustrated as a search key in a 
document level similarity calculation process. A process of calculating all the similarity to this 
search key in these media in an accumulated document about all the search keys for every 
media including text information, picture information, and speech information which are 
included in an illustrated document. 

A process of making into similarity of representation to a search key what has the highest 
similarity. 

A process of calculating average value of similarity of representation. 

A document level similarity setting process which sets up similarity in a document level of 

media. 

[0051]This invention (claim 55) is provided with the following. 

About each search key of two or more media in a document illustrated as a search key in a 
document level similarity setting process. A process of calculating all the similarity to this 
search key in these media in an accumulated document about all the search keys for every 
media including text information, picture information, and speech information which are 
included in an illustrated this document. 

A process of making into similarity of representation to a search key what has the highest 
similarity. 

A process of making into similarity in a level in a media document what has the highest 
similarity among similarity of representation. 
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[0052]This invention (claim 56) expresses each of structure information on a document 
illustrated as a search key, and structure information on an accumulated document as a tree 
with an order label (ordered labeled tree) in a similarity calculation process, Characteristic 
information of each media in each document is expressed as a tree with an order label with an 
attribute stored as an attribute of a node in a tree with an order label, By comparing an 
attribute and shape with a tree with an order label with an attribute expressing a tree with an 
order label with an attribute expressing an illustrated document, and an accumulated 
document, a similar retrieval process of setting up similarity of a compound media document is 
included. 

[0053]This invention (claim 57) about a tree with an order label with an attribute expressing an 
accumulated document with a node which has characteristic information which is the attribute 
of each node of a tree with an order label with an attribute which expressed an illustrated 
document in a similar retrieval process, and similar characteristic information as an attribute. A 
process of setting up similarity of a compound media document from a difference in structural 
physical relationship of this node is included. 
[0054]This invention (claim 58) is provided with the following. 

A process of considering that document structure is a tree with an order label, and evaluating 
and mapping this characteristic information on multi dimensional vector space in a tree shape 
comparison process with an order label based on characteristic information about this tree with 
an order label. 

A process of setting up distance on vector space as similarity of a document. 

[0055]This invention (claim 59) calculates similarity of a document by evaluating and using 
position information on a name and a node number of each node of a tree with an order label, 
or each node as characteristic information about a label tree with an order. As characteristic 
information, if this invention (claim 60) is text information, If it is the frequency of occurrence of 
a concept which a descriptive content of a text expresses, or each word, and picture 
information, If it is hue of a picture, chroma saturation, luminosity, color arrangement, and 
speech information and is strength of a sound, a melody, and structure information, let shape 
of a tree when a tree with an order label expresses document structure, a label name of a 
node, link information, etc. be the characteristic information from which it is extracted from a 
component of a compound media document. 
[0056]This invention (claim 61) is provided with the following. 

Information and structure information on media containing a text, a picture, and a sound which 
constitute a document illustrated considering similarity of an accumulated document to an 
illustrated document as a search key in a document comparison process. 
A process of setting up an evaluation value based on a similarity decision result for every 
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component with information on media and structure information containing a text, a picture, 
and a sound which constitute a document accumulated as whole compound media document 
similarity. 

A process of carrying out ranking by arranging similarity of an accumulated document in a 
descending order, and judging similarity. 

[0057]This invention (claim 62) is provided with the following. 

A process of setting up similarity for every component of a compound media document in a 
similarity setting process. 

A process of calculating similarity for every characteristic information of text information 
included in a document, picture information and speech information, and structure information. 
A process of making into similarity as the whole compound media document what applied a 
value of dignity to similarity and took linear combination. 

[0058]This invention (claim 63) includes a process of using the similarity of each component 
itself, or a thing which multiplied dignity given to similarity of each component by user as an 
evaluation value based on a similarity decision result for every component of a document, in a 
similarity setting process. As mentioned above, in this invention, it becomes possible to 
determine a unit of components, such as information on media, and structure information, 
which should compare between an illustrated document and an accumulated document by 
analyzing the syntax of a given document. 

[0059]Search based on the contents of the document or information on the logical structure is 
enabled by a document being characterized by extracted characteristic information. It becomes 
possible [ not only text information but picture information, speech information, structure 
information, etc. ] to use as a part of search key. It becomes possible to access a document at 
high speed by creating an index from text information accumulated, picture information, 
speech information, structure information, etc. to a document having contained them. 
[0060]Since the contents of characteristic information, such as text information of a document 
and picture information, can be checked on a display, it is easy to input a compound media 
document including characteristic information which a user meant as a search key. For every 
characteristic information included in a compound media document inputted as a search key, 
similarity is calculated and an evaluation value based on them is calculated. For example, for 
every characteristic information of a document, similarity is calculated and similar retrieval of a 
compound media document which also used the similarity of a picture, speech information, 
and structure information as a search condition in addition to text information becomes 
possible by calculating what applied, added and united a value of dignity with them as 
similarity in a document level. 
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[0061]Since similarity is calculated for every characteristic information in a document, a 
similarity calculation method that only similarity calculation methods of picture information differ 
can be adopted concerning each similarity calculation method, and replacing selectively can 
carry out easily. 
[0062] 

[Embodiment of the lnvention]As a component of a compound media document, as shown in 
drawing 3 , there are text information, picture information, speech information, structure 
information, etc. Hereafter, the similar retrieval in the compound media document concerned is 
explained. Drawing 4 shows the composition of the similarity retrieval device of the compound 
media document of this invention. 

[0063]The similarity retrieval device of the compound media document shown in the figure 
comprises the compound media document input device 10, the search condition input device 
20, the search condition input section 30, the compound media document analyzing part 40, 
the characteristic information extraction part 50, the accumulating part 60, the memory 70, the 
document comparing element 80, and the display 90. The compound media document input 
device 10 inputs a document including text information, picture information, speech 
information, and structure information. 

[0064]The search condition input devices 20 are pointing devices, such as a mouse used for a 
user's input, a keyboard, etc. The search condition input section 30 makes a user input a 
document filing name from the keyboard which is the search condition input device 20, or. The 
compound media document inputted as a search key by making a mouse operate it, and 
making it click on the icon of a document, or making the document obtained by the last search 
results click with a mouse is acquired. The compound media document which serves as a 
search key for searching a compound media document in detail is illustrated. A search key is 
illustrated by specifying the document filing name to illustrate or clicking on the icon of the 
document to illustrate on a display with a pointing device etc. It is possible to specify the 
portion than to which a user wants to attach greater importance to similarity, when illustrating a 
document, and it is possible to change the value of the dignity which shows the degree to think 
as important suitably, and to input it to the characteristic information of the portion which wants 
to think similarity as important. At this time, the document number k returned as the value and 
search results of the dignity of which portion in a document to think the similarity as important 
is acquired from a user. Or the default value of a system is used. 

[0065]The compound media document analyzing part 40 analyzes the syntax of the compound 
media document input device 10 or the document given from the search condition input section 
30, and detects the component of documents, such as text information, picture information, 
speech information, and structure information. The compound media document analyzing part 
40 analyzes the document inputted using the purser (parser: syntax analyzer) of SGML or 
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XML, and detects the component of documents, such as text information, picture information, 
speech information, and structure information, from a document here. 
[0066]The characteristic information extraction part 50 extracts the characteristic information 
expressing the feature of the component of documents, such as text information, picture 
information, speech information, and structure information. For example, if the concept etc. 
which the descriptive content of a text expresses if it is text information are picture information, 
and the hue of picture information, chroma saturation, luminosity, color arrangement, etc. are 
speech information, characteristic information, such as strength of a sound and a melody, is 
extracted with ID of a document, and the element name and the information on an appearing 
position that characteristic information was stored. If it is structure information, it is considered 
as the characteristic information from which the shape (layered structure etc.) of a tree when a 
tree with an order label expresses document structure, the label name of a node, link 
information, etc. are extracted from the component of a compound media document. 
[0067]The accumulating part 60 accumulates the given document in the memory 70. The index 
to the document which included the characteristic information concerned from each 
characteristic information is created. By comparing the characteristic information of the 
illustrated compound media document and the compound media document accumulated in the 
memory 70, the document comparing element 80 asks for similarity, and outputs the high thing 
of similarity. The similarity as a compound media document should calculate the evaluation 
value based on each similarity calculation results, such as text information, picture information, 
speech information, and structure information. Concerning [for example, ] similarity, such as 
text information, picture information, and speech information, Based on a multi dimensional 
vector space model, map each characteristic information to up to multi dimensional vector 
space, and if the distance for two points of the characteristic information of the illustrated 
document on multi dimensional vector space and the characteristic information of the 
accumulated document is near, It is possible to adopt the approach of setting up so that 
similarity may become high. It is also possible to carry out ranking by arranging the similarity of 
the accumulated document in a descending order, and to judge similarity. 
[0068]Hereafter, the operation in the above-mentioned composition is divided into a compound 
media document storing phase and a compound media document-retrieval phase, and is 
explained. Drawing 5 is a flow chart of the compound media document storing phase of this 
invention. 

Step 101 A compound media document is first inputted from the compound media document 
input device 10. 

[0069]Step 102 The compound media document analyzing part 40 analyzes the syntax of the 
compound media document inputted from the compound media document input device 10, and 
detects the component of documents, such as text information, picture information, speech 
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information, and structure information. 

Step 103, next the characteristic information extraction part 50, About document constituents, 
such as text information, picture information, speech information, and structure information, if it 
is text information, for example, If the concept etc. which the descriptive content of a text 
expresses are picture information and the hue, chroma saturation and luminosity of a picture, 
color arrangement, etc. are speech information, characteristic information, such as strength of 
a sound and a melody, is extracted with ID of a document, and the element name and the 
information on an appearing position that characteristic information was stored. The 
processing concerned is repeated several minutes of all the components. 
[0070]Step 104 The accumulating part 60 creates the index to the given document and the 
document which included the characteristic information concerned from each characteristic 
information, and stores it in the memory 70. Next, operation of a compound media document- 
retrieval phase is explained. Drawing 6 is a flow chart of the compound media document- 
retrieval phase of this invention. 

Step 201 the search condition input section 30 makes a document filing name input from the 
keyboard which is the search condition input device 20, or. The compound media document 
inputted as a search key is acquired by making a mouse operate it, and making it click on the 
icon of a document, or making the document obtained by the last search results click with a 
mouse. At this time, the value of the dignity of which portion in a document to think the 
similarity as important, and the document number k returned as search results are acquired 
from a user. Or the default value of a system is used. 

[0071]Step 202, next the compound media document analyzing part 40, The syntax of the 
compound media document inputted from the compound media search condition input section 
30 is analyzed like processing of a compound media document storing phase, and the 
component of documents, such as text information, picture information, speech information, 
and structure information, is detected. 

Step 203 the characteristic information extraction part 50 like a compound media document 
storing phase, The characteristic information of document constituents, such as text 
information, picture information, speech information, and structure information, It extracts with 
ID of a document, and the element name and the information on an appearing position that 
characteristic information was stored, and characteristic information is extracted about the 
component of documents, such as text information of the illustrated document, picture 
information, speech information, and structure information. The processing concerned is 
repeated several minutes of all the components. 

[0072]Step 204 The document comparing element 80 compares the characteristic information 
of the illustrated document with the characteristic information of the document accumulated in 
the memory 70, calculates similarity for each characteristic information of every, and calculates 

httn://www4.ir)d1.inr)it.eo.iD/cei-bin/tran web cei eiie?atw u=http%3A%2F%2Fwww4.i... 12/24/2008 



JP,2000-148793,A [DETAILED DESCRIPTION] 



Page 19 of 25 



the evaluation value based on those calculation results as similarity as a compound media 
document. The calculation method of similarity is mentioned later. 

Step 205 The document comparing element 80 arranges similarity in a descending order, and 
chooses from an index the document of top k affairs which the user demanded as a high 
document of similarity. 

[0073]Step 206 It displays on the display 90 by making the high document of the selected 
similarity into search results. Next, how to ask for the similarity in the above is explained. 
Drawing 7 is a flow chart at the time of performing document comparison for asking for the 
similarity of this invention (the 1). 

[0074]Step 301 The document comparing element 80 acquires the characteristic information of 
the document (search condition) inputted from the search condition input section 30, and the 
characteristic information of the compound media document inputted from the accumulating 
part 60 from the characteristic information extraction part 50. 

Step 302 When characteristic information is structure information, it shifts to Step 303, and 
when that is not right, it shifts to Step 304. 

[0075]Step 303 The document comparing element 80 calculates the compilation distance 
between the two trees concerned by considering that the structure information on the 
characteristic information of a search condition and the characteristic information of a 
compound media document is a tree, and shifts to Step 306. The methods of calculating the 
structural physical relationship between nodes, such as multi-dimensional-vector-izing a tree 
and carrying out [ which it is on multi dimensional vector space ] distance calculation, are also 
possible. 

[0076]Step 304 The document comparing element 80 calculates the distance on multi 
dimensional vector space. 

Step 305 Distance chooses the minimum thing as a representative among characteristic 
information of the same kind. 

Step 306 The similarity in a document level is calculated. 
[0077]Step 307 Similarity chooses a high document from an index. 
Step 308 The selected document is outputted to the display 90. The thing of the above- 
mentioned similarity calculation searched for for similarity as follows as law on the other hand, 
for example is possible. 

The 1st similarity calculation method : (1) Information and structure information on media 
containing the text, picture, and sound which constitute the document illustrated as a search 
key, The evaluation value based on the similarity decision result for every component with the 
information on media and structure information containing the text, picture, and sound which 
constitute the document accumulated in the memory 70 is set up as whole compound media 
document similarity. Here, with the evaluation value based on a similar decision result, the 
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similarity of each component itself or the thing which multiplied the dignity given to the 
similarity of each component by the user is used. 

[0078](2) The 2nd similarity calculation method : calculate the similarity of text information by 
finding the distance on the multi dimensional vector space of the characteristic information of 
the inputted text information, and the characteristic information of the text information 
accumulated in the memory 70. The similarity of picture information is calculated by finding the 
distance on the multi dimensional vector space of the characteristic information of the inputted 
picture information, and the characteristic information of the picture information accumulated in 
the memory 70. 

[0079]The similarity of speech information is calculated by finding the distance on the multi 
dimensional vector space of the characteristic information of the inputted speech information, 
and the characteristic information of the speech information accumulated in the memory 70. In 
each above-mentioned information, what has a small distance on multi dimensional vector 
space is calculated as what has high similarity. 

[0080](3) similarity calculation method [ of ** a 3rd ]: - the above-mentioned method of (2) 
again - in addition, when two or more same media exist in the same document, similarity sets 
to the similarity of representation [ the highest thing ]. For example, as shown in drawing 8 , in 
the document containing two or more picture information, into the accumulated document, 
similarity with two or more picture information is calculated, and similarity sets up the highest 
thing as similarity of representation in it about the picture information which exists in the 
document illustrated as a search key. This is performed about all the picture information which 
exists in the document illustrated as a search key. 

[0081 ]ln drawing 8 , it asks for each similarity of the picture A in an illustration document, and 
the picture a, b, and c in a stored document, and similarity makes the highest thing (for 
example, the picture a) the similar picture in the stored document to the picture A in an 
illustration document. It asks for each similarity of the picture B in an illustration document, and 
the picture a, b, and c in a stored document, and similarity makes the highest thing (for 
example, the picture c) the similar picture in the stored document to the picture B in an 
illustration document. 

[0082](4) The 4th similarity calculation method : the structure information of the characteristic 
information considers that document structure is a tree, calculates compilation distance 
required in order to change into another tree from one tree, and if compilation distance is 
small, it will set it up again so that similarity may become high. Compilation distance is 
computed from cost required to perform [ the number of times which performed editing 
operation called insertion of a required node, deletion of a node, and change of a node name 
when changing a tree, and ] those editing operation. Thereby, what has compilation distance 
possible calculating similarity and small is calculated as a high thing of similarity. 
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[0083]The 5th similarity calculation method : (5) Text information, picture information, speech 
information, The value of the dignity of which portion in the document which calculated 
similarity, such as structure information, respectively and was acquired by the search condition 
input section 30 to think the similarity as important, Or based on the default value of a system, 
the value of the individual dignity given to each of similarity, such as text information, picture 
information, speech information, and structure information, is applied, and linear combination is 
taken. What took this linear combination is equivalent to the similarity as a compound media 
document. 

[0084](6) In 6th similarity calculation method:, next the document comparing element 80, when 
calculating the similarity of a compound media document, After calculating the similarity based 
on the characteristic information of the text information included in a document, picture 
information and speech information, and structure information and performing the 1st-step 
selection based on the similarity calculation result based on the characteristic information of 
structure information, Let similarity based on the characteristic information of the text 
information and picture information which are contained in a document, or speech information 
be the similarity as the whole compound media document. 

[0085]This method is explained in detail below. Drawing 9 is a flow chart at the time of 
performing document comparison for asking for the similarity of this invention (the 2). 
Step 401 In the characteristic information information extraction part 50, the characteristic 
information over the inputted search condition is inputted. 

[0086]The similarity of step 402 text information, It calculates by finding the distance on the 
multi dimensional vector space of the characteristic information of the inputted text information, 
and the characteristic information of the text information accumulated in the memory 70, or 
searching for the difference of the value obtained from the frequency of occurrence etc. of the 
characteristic information of the inputted text information, and the characteristic information of 
the accumulated text information. 

[0087]The similarity of picture information is calculated by finding the distance on the multi 
dimensional vector space of the characteristic information of the inputted picture information, 
and the characteristic information of the picture information accumulated in the memory 70. 
The similarity of speech information is calculated by finding the distance on the multi 
dimensional vector space of the characteristic information of the inputted speech information, 
and the characteristic information of the speech information accumulated in the memory 70. 
What has a small distance on multi dimensional vector space, and what has a small absolute 
value of the difference of the value obtained from the frequency of occurrence etc. are 
calculated as what has high similarity. 

[0088]What [ compilation distance required in order that structure information may consider 
that document information is a tree and may change it into another tree from one tree among 
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characteristic information is calculated for ], It is possible to calculate similarity by evaluating 
and multi-dimensional-vector-izing wooden characteristic information, and finding the distance 
on multi dimensional vector space etc. What has small compilation distance, and what has a 
small distance on multi dimensional vector space are calculated as a high thing of similarity. 
[0089]Similarity, such as step 403 text information, picture information, speech information, 
and structure information, is calculated, respectively, Based on the value of the dignity of which 
portion in the document acquired by the search condition input section 30 to think the similarity 
as important, or the default value of a system, the 1st-step selection based on the similarity of 
structure information is performed. 

Step 404 Similarity, such as text information of the document which remained after performing 

the 1st-step selection, picture information, and speech information, is equivalent to the 

similarity as a compound media document. 

[0090]Step 405 Similarity chooses a high document from an index. 

Step 406 The selected document is displayed on the display 90. 

(7) In similarity calculation method [ of ** a 7th ];, next the document comparing element 80, 
when calculating the similarity of a compound media document, After calculating the similarity 
based on the characteristic information of the text information included in a document, picture 
information and speech information, and structure information and performing the 1st-step 
selection based on the similarity calculation result based on the characteristic information of 
text information, picture information, or speech information, Let similarity based on the 
characteristic information of the structure information included in a document be the similarity 
as the whole compound media document. 

[0091]This method is explained in detail below. Drawing 10 is a flow chart at the time of 
performing document comparison for asking for the similarity of this invention (the 3). 
Step 501 The characteristic information of the inputted search condition is inputted. 
The similarity of step 502 text information, It calculates by finding the distance on the multi 
dimensional vector space of the characteristic information of the inputted text information, and 
the characteristic information of the text information accumulated in the memory 70, or 
searching for the difference of the value obtained from the frequency of occurrence etc. of the 
characteristic information of the inputted text information, and the characteristic information of 
the accumulated text information. 

[0092]The similarity of picture information is calculated by finding the distance on the multi 
dimensional vector space of the characteristic information of the inputted picture information, 
and the characteristic information of the accumulated picture information. The similarity of 
speech information is calculated by finding the distance on the multi dimensional vector space 
of the characteristic information of the inputted speech information, and the characteristic 
information of the accumulated speech information. What has a small distance on multi 
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dimensional vector space, and what has a small absolute value of the difference of the value 
obtained from the frequency of occurrence etc. are calculated as what has high similarity. 
[0093]What [ compilation distance required in order that structure information may consider 
that document structure is a tree and may change it into another tree from one tree among 
characteristic information is calculated for], It is possible to calculate similarity by evaluating 
and multi-dimensional-vector-izing wooden characteristic information, and finding the distance 
on multi dimensional vector space etc. What has small compilation distance, and what has a 
small distance on multi dimensional vector space are calculated as what has high similarity. 
[0094]Similarity, such as step 503 text information, picture information, speech information, 
and structure information, is calculated, respectively, Based on the value of the dignity of which 
portion in the document acquired by the search condition input section 30 to think the similarity 
as important, or the default value of a system, the 1st-step selection based on similarity, such 
as text information, picture information, and speech information, is performed. 
[0095]Step 504 After performing the 1st-step selection, the similarity of the structure 
information on the remaining document is equivalent to the similarity as a compound media 
document. 

Step 505 The high document of similarity is chosen from an index. 
Step 506 The selected document is outputted to the display 90. 

(8) The 8th similarity calculation method : when calculating the similarity of a compound media 
document and two or more same media exist in the document illustrated as a search key, set 
up the similarity in the document level of the media concerned. For example, by (8), although 
above (3) described the case where two or more same media existed in the document of the 
accumulated retrieval object, when three different pictures in the document used as a search 
key exist, the point how to set up the similarity of the picture in the document used as a search 
key is explained. 

[0096]When calculating the similarity of a compound media document and two or more same 
media exist in the document illustrated as a search key, based on two kinds of examples which 
set up the similarity in the document level of the media concerned, the case where picture 
information exists in a document is described. 

** When setting up the similarity in the document level of media, about each search key of two 
or more media concerned in the document illustrated as a search key. All the similarity to the 
search key in the media in the accumulated document is calculated, When similarity makes the 
highest thing the similarity of representation to a search key, calculates the average value of 
the similarity of representation and considers it as the similarity in the document level of media, 
the picture a is acquired as a similar picture in the stored document to the picture A in an 
illustration document shown in drawing 8 . The picture c is acquired as a similar picture in the 
stored document to the picture B in an illustration document. The average value of the 
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similarity between the picture A and the picture a and the similarity between the picture B and 
the picture c is calculated, and the value is set to the similarity of the picture information in the 
document level between an illustration document and a stored document. 
[0097]** When setting up the similarity in the document level of media, about each search key 
of two or more media concerned in the document illustrated as a search key. All the similarity 
to the search key in the media in the accumulated document is calculated, When similarity 
makes the highest thing the similarity of representation to a search key and makes what has 
the highest similarity the similarity in the document level of media among the similarity of 
representation, the picture a is acquired as a similar picture in the stored document to the 
picture A in an illustration document shown in drawing 8 . The picture c is acquired as a similar 
picture in the stored document to the picture B in an illustration document. What has the 
highest similarity (for example, similarity between the picture A and the picture a) is set up as 
similarity of the picture information in the document level between an illustration document and 
a stored document among the similarity between the picture A and the picture a, and the 
similarity between the picture B and the picture c. 

[0098]Next, when performing similar retrieval of a compound media document, the example 
which expresses each of the structure information on the document illustrated as a search key 
and the structure information on the accumulated document as a tree with an order label 
(ordered labeled tree) is explained. Drawing 1 1 is a figure for explaining expressing the 
compound media document of this invention as a tree with an order label with an attribute. 
[0099]When performing similar retrieval of a compound media document, it expresses as a 
tree with an order label with an attribute (tree which extended the tree with an order label) 
which stored the characteristic information of each media in each document as an attribute of 
the node in the tree with an order label concerned, The similarity of a compound media 
document is set up by comparing an attribute and shape with the tree with an order label with 
an attribute expressing the tree with an order label with an attribute expressing the illustrated 
document, and the accumulated document. 

[0100]When setting up the similarity of this compound media document, based on the 
characteristic information about a tree with a label with an order, on multi dimensional vector 
space, the characteristic information concerned is evaluated and mapped and the distance on 
the vector space concerned is set up as similarity of a document. Evaluation of characteristic 
information shall evaluate the name (label name) of each node, and a node number and the 
position information on each node. 

[0101]As mentioned above, even if a user does not know detailed document structure, search 
of the document which also used structure information in addition to text information can carry 
out. The information and structure information on a picture or a sound can also be included in 
a part of search key besides text information, and similar retrieval of a document can be 
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performed. The search condition input section 30, the compound media document analyzing 
part 40, the characteristic information extraction part 50, the accumulating part 60, and the 
document comparing element 80 which are shown in drawing 3 are built as a program, It 
stores in portable storages connected to the computer used as a similarity retrieval device of a 
compound media document, such as a disk unit, a floppy disk, and CD-ROM, and this 
invention can be easily realized by installing, when carrying out this invention. 
[0102]Change and application are variously possible for this invention within a claim, without 
being limited to the above-mentioned example. 
[0103] 

[Effect of the lnvention]As mentioned above, according to this invention, even if a user does 
not know detailed document structure, the document also using structure information other 
than text information can be searched. The information and structure information on a picture 
or a sound can also be included in a part of search key besides text information, and 
resemblance of a document can be searched. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the 
original precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 

[Brief Description of the Drawings] 

[Drawing 1] lt is a figure for explaining the principle of this invention. 
[Drawing 2] lt is a principle lineblock diagram of this invention. 

[Drawing 3] It is a figure for explaining the compound media document of this invention. 
[Drawing 4] lt is a lineblock diagram of the similarity retrieval device of the compound media 
document of this invention. 

[Drawing 5] It is a flow chart of the compound media document storing phase of this invention. 
[Drawing 6] lt is a flow chart of the compound media document-retrieval phase of this invention. 

[Drawing 7] It is a flow chart at the time of performing document comparison for asking for the 
similarity of this invention (the 1). 

[Drawing 8] It is a figure for explaining the method of similarity setting out in case two or more 
same media exist in the same document of this invention. 

[Drawing 9] It is a flow chart at the time of performing document comparison for asking for the 
similarity of this invention (the 2). 

[Drawing 10] lt is a flow chart at the time of performing document comparison for asking for the 
similarity of this invention (the 3). 

[Drawing 11 l it is a figure for explaining expressing as a tree with an order label with an 
attribute of the compound media document of this invention. 
[Description of Notations] 

10 A compound media document input means, a compound media document input device 
20 Search condition input device 

30 A search condition input means, a search condition input section 

40 A document analyzing means, a compound media document analyzing part 

50 A feature extraction means, a characteristic information extraction part 
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60 An accumulation means, an accumulating part 
70 Memory 

80 A document comparison means, a document comparing element 
90 An output means, a display 



[Translation done.] 
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£LT, &#i$gfg©gM?©t>©, ifcH\ &#§J& 
*7»©*K«WifflI. 

[»««4 3] mum, bmhb, xm<DP<D 

*iHit-* *§*r-* fcStrtfWWifr 6*Uff * ft * 

Afc^LfcElt&ttTfcoT, 20 
4*SftfctMB*&;<T-f7£«fttf* AftSftfc&H 

fl!)«j«fr61«t1MI*Jlllffit*^1IMWffl^* 
xt, 

mstTj7X9*m&fttLT\j]-£mmm 

kt>7zH>Zt, 30 

KriEHI#aicWI«ftTV^ttEll&^ T-f 7£«© 
ItfMNBfc. A**ftfcl9IESU^feft*l(lESW»llf^ 

■feXTttaSftfcWWHIfrB 2"3©Ji£*T'f 7£« 

z^tzztmmttzmstTjTicmmmm 
cn*S4 4] mmmx-h-fv*.*^ 40 

z?j3*gmmm , 7'v*zt. 
ztiTM*T<( 7%s<D%mm$!tiitz>mm 
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mmmt, mmmm.®m&7ti*xxmz 
ntzmmmmc**), m&m*T<(7zmttm 

mmmmxii^xn^xxmmm^-t bxmt 
ftfeA»*«auT^2r*xh«a, mm®. 
$mvmmt, mffizmmtwrntix^m 
mm*T47%9*ffl8.Lx^z>T*zhm. a® 
m. ^mmmmntmmmmum 
fg&gtm-Ktmm. m^^r^7tsm<Dm^ 
&tLxm.tzm®mM7v*xzttsmm4 3 
Amm<om^T^7tmmwm^u^h 

mmimm<Dmmy\m%t lx. 
mmmztirctsvT** ymtmEmmw 
mtix^zffi£tm<DT*z h««©sflatt«j£«j& 
t, mmttiitxmmwmt. mmmemm 
znx^zttmmntmommzst, mm 
**ftfc:»©**««fceae*#©fc«a?ftT^s 

A«©^«ai:©SfilttW^«fc, »t^7c3» 

7^mmm7v?7i*zmLtcmm#. 0 

mm 4 7 ] WSfilASSya -bXli, 
WfbWfc^SftSf+XHWI, BftflMW&TIII 

afti*fcs*©#*»it, wea*k-3fet©*^ me 

«*yr^7»^fcbT©«H*tr*WKdllflI 

4 8 ] MIBiSitSyn 
P3-X««ffcra-^7'Y7* t fflS#«E-r*«^fc^ ME 

0<j^?ftfcx»^fc^^n^r*^ k a$u 
ts*rj7m£x<D]m*-fco^x, wmznrct. 

»*©*r-f7fcfctf*B8«*-fc*tt*£T©Sffil 

»-r*f^i©a<KiSEi:i-*f^aftifla*^yD-bx%^ 

tflf3»B4 3Rtf4 4EK©H^^ty 7XS©SMHtt 
H?7d ^^fcffiW LfcEIIKf*. 

ME&iis*-£LT> IOBWS«ftfcX«©IWHWIfc 

sa?ftfex»©«i««©*ft^ft*, m^Mi 

* (ordered labeled tree) tbXmUb. %£BMmZ 

fttc^%mwc®mi¥7<frmt, tamwmz 
ctx\ mmmmQmmzwfetzm?^ 
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ttmmMfv -b TJt^tsmm. 4 4 mmm* r 

mm 5 0 ] mmy^mmmfv 
it. 

Bir*WIOT*WE»^fla«fctTHS-rs»* 10 
[H^5 1] iWHJHH[OHI^D*XH:, 

jit *g<o#mif$g£g-3< «ttt*tHm7n-b* 
t> 

«wwio»«i«ifc«^< m»mmmcMrs< % 

vmmicm-Kmmz. ^^r^rtmmt 20 
LT0)m®L&ttz'?n*zt*$tsn$.m4 mm® 

mm 5 2 ] mmmmfv 

f «fc»-3< »fl«tHWg«fcStJ< % 1 gPg@<DlJR 

zm^B-tzts 30 

tf&t%i§^\c m*TJ7<D$auo]sT-(Dm&m 

Lzmbfcmmfto 40 
mm 5 4 ] mm*g\s^mmtm7v*z 

T4 7<D&m*-\c-D^x, mmmzntcitwc 

R^T^7t^t*ij^*-fcjitr4^T©affii«* 

(ommtt^o^Tst. 50 
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* 7^»©^^^p ?7 A*1S$ LfcEHig 

r<(7(D&im*-\c"D^T, m&mtfttzzmw^ 

* r * 7\cmmm*-\zm%£x<omtmim 

tsmm 5 4 ek©*£* 7^ 7i««s{y^7 > D y 

mm 5 6 ] MIB^fKSItl/n-bXtt, 

m*-t Lxmztirzi&Qimmtwmtirc 
x^mmmztiznzm^mz* (ordere 

d labeled tree) t LT$0&U 

*T47mmmmmm¥7<Mi%*fci3Vz; 

m^ mm<om^^T^7tS(ommmfti^yL 
mm 5 7 ] mmmmm-fv -trxfct 

mm 5 s ] mm&7«jw**iewt&?v* 

±fcWMMWI*»ltfl:tTvy if yy-rsya-fex 
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K Bfe *^*^«y^7 f -f7©««Rtf«B!Ht«tO 20 

EKoa^^r^7x«offl««ai^o^i.*tsttt i 
[ft*® 6 2 ] tummmmzfvwt, 

Wlsa^r^75»0*«riSSI5S»i:. 
•f £7°P-fe7.^ 

Kraxwcss nsr*x Fit Mm *w« 

/cfefl!)*fiJffl-rsyp-bX*^tff|^4 4fE®©H£ 
7£»©^M^7W^£^Lfc!Eii® 

^©ffinftijwn 

[0 0 0 1] 

7m*mt5mms*T4 7^mwmi5 50 



30 
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[0002] 

fcLTlis £&?^;*#©StructureIndex EContent 1 
ndex ©fi»£t>* finfe t/ik "«* 

6©^rATftt, SGML£#^XML;£«&H©4i 

x»©«ai*Rri!BfctT^'>*. 

[0 0 0 3] F>'XrA©ConceptBase ^377 

77 h©VextSearch*if©^XrAT\ LT 

*» (r+Xh©*) ©fcjfcfcnregfcbT^S. 
[0004] 

8£*©75ftfe%«^^ 75W*«*fc LWWWaifc 

[0 0 0 5] -WW^1iZT*Zh&JM>*T<(7 
7#«©gGlfcfistfT?tfcv\> 

bTa^r^7i»3WW^?nSi:, m*-$nrc±w 
teSSftSf^XHUft B&^^llft JtjiffS 

4^©waitai:«i?nfcx«t^sn5r*xns 

ft H«W©«ft «WI«ft2©W«l«fc«- 

nwtKu r^x us $g©£i{«fi, B«fii©^H 
a, ^ia©siKfi> mmmmmm^m 

W-gtfet©*XSl^;l/T*©^ffilfifcL, !#©Mr 
Wflstscfc-p, r*xMIWWIHtttt*«ltfcJWtt 
1t^^ffl#fS©SlH14^S«tfc^{K^^Pjif« 

©aflatt*iabfca<K«»a ^tf^ftJis* r w 7 

X»©a«iaH5S»&tf«KRtf1i^^ 735«©^ 
[0006] 
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mmztefctzrtibQzm] m 1 it. mmommz 
mwtzfctbvm&Zo mm (h^d it. 
x him, ®«ua, xmvpcDT-ZT&zgpT- 

flm«vnt at 7/3) , a^^r-f7x»*«s? io 
7^6) , s^nT^5^i^isifii^^2(D#m 

t»c*-3^T 2 r-f 75W©affiUSE*WSg 

L UT7/7) , MHLfcil**T<f7X»*atirt- 
3 Uf?78) o 

[0 0 0 7] #S£BJ! («*«2) tt, ^^tfe^T> 
f!Iffl«*Wwt i:LTA^]Ls 

nrcmfrimmmwtmu mtnmz 20 
©otitis 1 <omm\Oi Qzsmmmtm 
m*o mm <mm3) it, m^/T^rmm 

#MSl/C^3r*xK B#> «***&;<t*7©« 
mh, B®> S**^*r*7©tt«&tf*Bfi1il 

[0 0 0 8] #%B£ at*E4) tt, flbftgSRttQSHH 
ttWJSBft l/t\ «9^?nfcA»Or*x HfSk, 30 

[0 0 0 9] #m 01*85) tt, ££*tV7£« 
[0 0 10] #8BJi 0m6) It, fe£*T<7*M 

<ommmn?m\Zs n-x»+icmi-*7V7tf 

XMiffg, IHMR& WJHBfcStMT^SO^T 

t-^v^T, wmztitc$9'p<DmT4 7\z 
mmtm^vzmz-icmzmmmtt 

5„ 50 
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[0 0 11] #£BJ§ 00*9(7) tt, 8S*T>r7:fc» 

vrnmrnwrmiL. m*-tLx, mztm 

£n IfiJ^'-Wft* (ordered labeled tree) tLX% 

[0012] mm am 8) it. mmnmm 
[0013] am 9) »s a^^T-i7X» 

JMS«fc«^< % 1 8UBiOjSR*frv\ x»t#$n 

sf^xnt mmm^mm.ommmi^-3 

[0 0 14] #%bj? ami 0) S> Jg£* r-r 7* 

mmtm-i < a«u»HBs*ica^ < % 1 ©pg g © 

<«ffitt*x ^^r-r7i»^i:bT©^{Kfifr 

[0015] *^ ami 1) it, m*T47% 

7<DXg\s^)]sT'(Dm®Mzwfet5o mm (mm 
1 2) a, *T<(7<D$&w\<-u<»mm*w£T5 

^T^7<D^^+-toi>T, w^«nfei»'t't-§' 

7«©£T©fcjg*-fco^-r\ sa^n^ci*^©^ 

[0016] *^ ami 3) it, *t47<dzm 

^X, 09*«nfcX«(ft^Sn*r*^HWB, B^ 

WW. %F®m%ts*r<{7m<D£x<Dm*-k-D 

^X, Wmtitz%9*<D&*T<(7lct5tfZ&.®M* 
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X'VfflmttZo [0 0 2 3] <mW (»#B2 1) Its m>*TJ7l£. 

[0017] mm mmi 4) it. a&*r^7£ mzfrmmmfetzmc zmvmjmmmm 

mommmzmu^ m*-tbxm*ztitc% mw7mmcg-3<wmf&tLx, 

IW¥7^M1Z* (ordered labeled tree) tLX^M 4*Snfct&£&tt£t>#fc&©Wi"3"5o 0 2 

u zftzm-mw&tT'i rmmmmv it, mm<DW,mmx-$>z 0 

^mxicmz;- wm&t Lx&MLtdMm [0024] mm mm2 2) it, 7*7. him, 

tzztx\ mtTsfrzmmmzmtZo mmmmmx&ox, $G*T<{7*g*xt>t 

[0018] mm m*m 5) it. a^r-^* J7*m\ttm ot. mtT^nm 

&mm?^m*<D&y- YvrnvLXh^wt \Ki*tiTzmmm%.m?%*mmi*m 4 0 

i/>t, j- Y<Dm&mmmmmfrt>m£* f 4 t , 7$mRmm mmm 5 0 vmu\ 

rxMmm&wtstio 20 2 nfci#mif«*sffitsss#s e 0 ms* 

[0019] am 1 e) it, mmmirn 7%mzmmt Lxkfitzm&ftxtim 3 0 

fi*fiSt*Bt, X«KB*iW9^/W***i:Jltt t> #»¥S6 ofcS*£ftT^sa£;<T-r7£fF0 

U IWWf^W**lc»r*»«1(l«lclS"^^T, #t1i*g£> X*«tifc«WefefM&*«IWf#94 or- 

£&tW h;l^M±tK1«MHI*»ttfl:tTvy if Wfrl/fclS*t«^^TWW» HaitJ¥S5 OX-SUBS 

y^U ^^h;l^BI±-eoffi(t«5«OSfilfii:b-C nfcW«1t«fcS^T2O0a£*T*7:»®S(H 

(n^g 1 7) a, iwft?^ **W3&r**»Jt«#a8 0 £*jt«&g8 0? 

ostomy - Kft^y - HOffiW«a*»«fl:l/Tf!l »*ffl*-r*lfl*¥«9 0 tZGtZo 

m -5 CI T*£«©g{«&£itg-f 3. [002 5] *^ (W#E 23)tt, ^^ftA*# 

[0020] *5sh (n*« 1 8) #ai9*i: l 30 g3 ofcfci^T, mmm*Lrz%m*m*-t l 

t, 7** h«*-e*ntf, 7*x h<DE^rt§^f TX*r*¥s%#*, ^mm^am#©5 ot^ 

ij >^it s=& if^, as* r-f 7X»©«nrasRft»6tt stm® 8 o (c6^t, xft^*t *»tH¥8-ejiii 

[0021] *8mi mm 1 9) a, «fiK*ws-r mmtntcm^Wim^^, %%*Tj7zmttk 

*x k mm, gpz-ststTj 7<Dmmmmm [0026] mm mm2 4) 

mm&m^^izm^mmm, m^^f^7 mmvmmnt, wmmMmnx^^ 

xszftmmthxmu wmtirrf&mm *T4 7$$*mf&Lx^%f**hm. mmn, 

[0022] mm mxm 0) it, mmzmt ts^< fws*, a^^rw 7*mm<»mm.t u 

s^u ittic#tn5r*x ha«x ffl^ifa^^ [0027] mm (mmz 5) a, mmm.m 
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tifc%m<DT*z hmtwwtmwmnx^z*. 

mm%B%t, mmzrczsoimmtwm® 

[oo2 8] mm am 2 6) it, mmmm 
m^x, itfc«n«f*7hfi«, wwnwf^ io 

u mmmicnwmzm, mmtoitw 

nmm%ts. mm (mm d it, zm&m 

%p*%ts*T>(7&oy£T<Dtiim*-ico^x, mm 
zntz*w$<D*T47\cmmmfi*-icmz>£. 
xvmmtmmu mmimt>nvt><Ditm* 
-icmzmvmmttzmmmmm*^ 20 

tr. 

[0029] mm mmz s) it. mmmm 
ios^x, ms—thx, mztircxmnimm 

ft* (ordered labeled tree) £LT^E|U #J^£ft 

mLtcMfty^Mtttmmitmzztx, * 
mmttso 

[0030] mm mmz 9) it, m^Mt* 30 
m&mm^x, mmmmmzmtm 
iz, $*mm*tmu mztittxm*m*fr 
zmznrc^mt*^mtzKtbiozm*s- 

m^tzmn, MWMtflF*fi3©fc£lKl:3X h 

[0031] (89*® 3 0) it, momim® 

#at, mmm<ommicm"3<m®Lmw&%ic 

mm*, m£*Tj7%w±fttLx<Dmimttz 
[0032] mm (m$m o it. mmmm 
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^fc^£ft*#§lttfg©ft&1flBfcg-3< 
iTO£s *&*7V7*»^i:l/C©MH*i:t* 

[0 0 3 3] mm (»*@3 2) 14, «Kfttr*#a 

*7tf«s#fttsii£ic. K^r-rr©»i^;PTf 

*£BJ§ (if £53 3) (4, £#l^^H£tm#Sk: 
■<7§©:£T©&if!*-fc:^T, s«?nfcit*o 

f«f^©afl«ftfr«¥afc, mmmmm 
mwtz^mt, ^r^7©i»i^;i/T-©s<Kfi* 

[0034] mm (m$g(3 4) (4, x«u^;m»k 

ta»#fiEt*^T-fr©#tt**-fc^Tv kw* 

it«**tr^T'f7«©^T©«ai*-fc'3^Tx m 

5 ftfc***©8* 7fc**t* WMMr-fcHt * 
£T©gfiHfi£frgt3#&^ »K*tf*fc«^t>© 
**»*Hc^*fta©*Hltfct*¥R^ f^a© 
Sflafi©^ *fc«fl^tfffi^fc©**T>f7©:S« 
U'Wl*©8fiUtfc 1 

[0 0 3 5] mm (HOT 3 5) «> MKlftttft^gl 
fcfc^T, ^*-£LT0J*2nfc£§©#§>gfflte 

g«?nfc;^©«^f$i©^n?n£iu*^/w>it 

* (ordered labeled tree) fcLT*£U ^tl^tl© 

x»*©#^r-f7©w«nia%»f7^wff*fc* 

JtS7- K©Jitti: LT«SWLfc*tt(**Wlf5^;W* 

"a^ T'f 7»©»fl«*IStetS«M*fl#a*# 
tr. 

[0 0 3 6] *8B£ (III&R3 6) Ifc jRflX&$#gt£ 
{***©£/- F©ffttV*Stttt1IHI2:StlKLfcttff 

%iLrcm&mm¥7^Miz*ic-D^x, ms-m 

*KSt5#a**tr. 

[0 0 3 7] #88J! (»^B3 7) tt, li^V^Wf 

^x, %m^>7 HWSIB±fcWMMIMI*»«il:tT 
■7'ylf>^t5¥ah, h;l^H±-e©BSI*X« 
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[0 0 3 8] mm (M$53 8) (is ||W*5^ 

mMtzc£vx8<Dmsi&zttwt&o mm cm* 

53 9) 14, #®ff8i:LT, T*X hit fBTfctttf* 
T*xh©Ea^^1J&*&¥R©lHS«& ■ 

[0 0 3 9] *gmi QR«fi4 0) {4, £§it&#©8 

ofcfcv^r, fl^sftfc£»wtt*tsreftfc£»© 

««©am*J^*fc»^<IWllHil^ a^^rw 

7X»4f*s««fcbTSS-rs?ai:, sasnfe* 20 
»©S<H**RJWc ft"** c t -vmtLWf u sms* 

[0 0 4 0] #8$ 01**4 1) tt, £Iffi(igfl£¥@! 

icfc^-c, »&^r^rx»©#i(MaaH5t, sous 
mmzmt, mmz.m<omm. mem 

[004 1] #|gB£ (11**4 2) {4, Aft(lfi6£#g 30 
fFfiUHfifcLT, £*8g*©»M**©fe©x £fc(4, 

^*>*ftto*«ffl-r*¥a**&. *£w «*B4 

3) f*XHt& MMi> Alffl©J»©r-*T 
*<MW*r-*fcin*, CD^Un-F^-tty^ftr- 

i>\ ktizntcmmmxfflrtzx&ffli?** 40 

fc, «£*TY7»*«BR*ftfcbTA**£3*iR 
7»<f7*»©<«MNli:, A£2nfcfclis£#££«8? 
■bXT«ifflSnfc<*ai8«fr6 2r3©ti^^T'f7X» 
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[0042] (mm 4 4) t4, tmmxir? 

tSAfr^&fKltattfiS^P-fexi:, 

5«©^it«^ttai-r5^#iiifatttii^p-bx^ 

aj7p*XTttaj2nfcA;rj;^#&1iSi:, fc^SffS 

7i»r^©^ffi(g^fi-»E-r5S{HfiH-iyp-fex^# 

tr. 

[0 0 4 3] #8BJ§ (If** 4 5) 14, SGlSW-gTfc 
■teXtefc^T, ^^#A^yp-tXT*^^-i:LT 

sa^r^7X»*««bT^«r*^Hfa, fi® 
f^, §?ft«Rtf«Bi«l8i:©«jaiyiW©a(ttttW 
^Jllcl^<i¥ffiii*, «£;<T*r7:fc«£tt©SIH 

[0 0 4 4] *^ (gg$g4 6) (4, ^(KgS^yP 

*£ftfc£»©T*x H»*teE1t¥®fc*«*mv 

[0045] mm 01**4 7) 14, mmmfefn 

•bXlcfc^T, ift^nST+XHIHI, stiffs 

st, tsa»H*fcs*©**£«\ mmtoTc&v 

*, S^^r^7X»^fti:tT©SffiUfti:-rs«^ft 
*fl:/p-feX*S&. *8BH (ff**4 8) (4, £«t 

K a®, ^F?*#tr*T-f-7£©£T©8S5*-fcO 

^t, wi*nfcX»*©^f f -f7ti3^sK««*- 

[0 0 4 6] *%B^ (f|**4 9) {4, WHSttJITa 

•bxicfe^T, &f!*-i:LT, 09^?nfci»©«PBi 
^;Wt* (ordered labeled tree) tLXBML. 00^ 
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[0 0 4 7] mm (!f*55 o) lis my^frtt* 

IBfcffSff-afcHRfc, KWKt^*(T^Ote£B43 10 
[0 0 4 8] (H*55 1 ) & SmU-SEyo 

MU&ttZ'fttZtZfStf. 20 
[0 0 4 9] *58W 01*55 2) tt % S&fifr^P 

[0 0 5 0] *%W (11*55 3) tt, g&gtm/p 

*r*7tf8»#a-r*J§si^ R^fYroitw 
%$tSo mm mm 5 4) a, jmi/^^tt 

«*£tr*7^7S©£T08SM t -fcO^ , tx Sfg£ 

<D%m*-fctitzm<Dmmttz>7vtzt, 40 

[005 1] mm (H*55 5) tt« Xt^MSffil 
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Cr. 

[0 0 5 2] *?£0J§ (11*55 6) H\ 8<«gfrl:/P 

■txtfcv^T, ^*-£LT0j*2ftfc£*©#t3tii 

ttf* (ordered labeled tree) fcLT«SU ^tt^ 

ft* «* 5^W**fcOKttfcJ5tt*Jtttt 5 c 
T\ «^^r-f7X«©«filfi%»S1-««ffit«ai^P 

[0053] mm 01*55 7) a, siH^yp-b 

F©l^Hftffi«IB«©f&gfrSa£* 7^ 7;SC#©S 
[0 0 5 4] 01*55 8) a, l»5^ttt 

[0055] mm (0*55 9) a> immtf 

F©£fK>/- F»*£7- F©ffi«««*afflfl:tT 

fust- 5ct-px«©a(K«*tni-rs. *aw cm* 
56 0) a> wm&tLx, r+xhii^T*^ntfs 

^it^T'Sna, ^©ssn^prf, *§jiff8T*&n 

7»©«Mssis^6atfisti*«F«*at-r«. 

[0 0 5 6] #8$ (11*5 6 1) tt> £#it&7P-fe 

H^> ^%#tf^T-f7©faRtf«IHt«i:©^ 
7XS^«fHfii:bTig«-rsyp-bX^, se?n 

** W3£tS y a -fe x i: *-&tr. 
[0 0 5 7] (||*5 6 2) ti, m<R^^7p 
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■b*fcfc^T, ££*T<(7jc9<D&Mf8mk&lc, ft 
fc, ft<M***rJW3:/a-feXi:, ftfcl£fcl*©tt* 

[0058] ami 6 3) a, mmm.-7v 
-nwrnnLx, mfmmvmmziD^ tit 

£T\ 0!i*«nfc*«i:»a*tifei»oiai-eJt«*fT 
[oo5 9] tks mhtzmmt 

[0 0 6 0] Sfc. f^^WlTXfOf+Xhft 
& WMI*4£©^««©fl«»BT?**©"^ 

^Slc, ftflHfcfcfrSU *ti6IOIt3<ff«i*tHI 
**. Wittf, ^©ft&HISSfc. ftfcttfctWU 3 

/PTCftflMfcfc bTtWt 5 c fcT\ hfflRUft 

[oo6 1] tare t&*<mmmtc, mmm 

[006 2] 4 

[0063] EBfc**a£*7V7:**©ftffi(8fts 
■a, a^^r^7X«x*««i o, «gR^x^St 

■ 2 0, ttsKfeftX*»3 0, «^r^7i««fK» 
4 0, !|fmii«affi^5 0, S«gi56 0, **'J 7 0, 
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T'f7i«x^s«i ottx r*xHtfB> gHitt& 

[0 0 6 4] fcfis&ftX*igfi2 Otis fflffl#tfX*© 

#— H97ft«. 83?*ftXa»3 0lk 
«SNfeftX*aelI2 0T»fcS*-*-FfrM»©77 

-oi/fcfcXrtstffctK v^x%jftff?*TX»©7'r 

«£v77T? U 7 ^«-&*C LTX7J 
10 tS«^r-f7X»*«fltS. Bb<tt* t£*r 
-f7A»*«a?t«fcft©«ai*-i:ft5a^^T-f7 

x»*fi^r*. w^t*i»©7 7>r;i/45*jsjsi/fc 

»K 0O^-rsXt©7'i'3>*#Wy^-fvyr^WX 
£ iffc £ D f<f *7l^±T* 'J » *"T3 C T*$iif* 

-*0H*t*. f fc> *»*w^r*BicN *w##ft 

«tt*«1BtfcvH»»*»r*ci:*ing»c*!), ft 

v^^-rs*ofi*ai«sbTX*"r* c tmmx 

20 fc^5S*©ffli:ttll@tg*i:LTSai-r*X«»k%f!J 

[0 0 6 5] «£*r-r 7*»«Wf»4 0ttx «£*T 
^7i#X^SHl OSfctt* «Sl*ttX*l»3 0^5 
4A6tifcX»©«XIWf*fiV\ Bft 
m ^1ffS, *JIHf«ftJf©i*©««Hil6*«ittJ 
?£o S^T^XWSHrSMOtt, CCT% SGM 
L-^XML©^— 9" (parser : $£1S?#t7W5A) * 
ffl^TX*snfcX»%SP*rU *«frP>r*Xr-ti! 

[0066] mnmtB»5 ott, vm. ffi 

flHMI, tJSfi, «Bi1ta42©XS©«^5©1# 
«*»RLTV*Sl«MM*ttart-*. w*tf» r^T, 

k isaa c>f+xh ©Earts*^t«gA ami 

^if^e.w©^n^^DT'fftH©^isif^, #® 
t»wia«?nTv^x»©iD, iwyh^ws 
tfeH©it«i:«icttm-rso ssfc, miftft&« 
40 «jB%««57^w**TSibfca^©*©«« (B 

2*a^^r-r7*«©ttdasR^6«mj?n*i*afli 

[0 0 6 7] g^gpeott, 4*6nft*»*^*'J 7 

oicta-r^c tit. mmm^mmmz^ 

ftffilS***, ftffi«©«v^t©%ia*T*. Jt£*r 
50 ^7i»fcUT©ftfl«tt, T+XUlWk WW**, 
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\,Vcfm®*mLTz$><DttZ>o mtt, r*7hlf 
18, IM ^1PB£H©gf«gfc:$Lm 

7 hfr&ffl±'W; £#7C^7 Wl/£PI±© 

s. mztirczmmimmmfc&^zct 
T*imm* u mmzmt s c t mm%>z><, 10 
[o o e 8] wt> m<Dwm^mmmm*T 

Xf 7 1 1 0 1 ) £t\ r -r 7£»A£g« l 

[0 0 6 9] 7.T771 0 2) ®&*T47tmffli 
B®jf?8, SfMIWL fl9B1H»i:2!«>£*Ottia8K« 20 

7 7 i o 3 ) mtwmffltf 5 o r+ 

XHS, BMffft WIMIh «BMMIftH©*««l 

©fssrts^-rsfcftftH, Bf&itf8&sB«i©£ii j e> 
* 0 3WM%r*T©#wi»i8©a^D«r. 

[0 0 7 0] 7.T771 0 4) maP6 0(i, 4^6 30 
fc*S'\©SK3l*f&*U **'J7 0fctt»*3. # 
06tt, *^©a-&^rw7XS«^7x-X©7a 

xf 7 72 o i ) m^x-hu 3 o a, mmx 

*T^7;£»*Ef#TS. C©i#, £»*©£©§?#© 

XrA©7^7*;l/htt*f!lJ§-r3o 
[0 0 7 1] 7T772 0 2) *fc, «^T-f7i 

ffftgtfrgMOtt, ^^r^73»Sa7x-X©®S 

mm. m*T4 7Wimx-huzoftt>\-h-£ 

tifcm£*T<(7$g<DM$Mmft<<\ r*7Ht 
& MHIfA ^1ff8, ®&1tl8ft£©:fc#©«tJSg 

msr&aj-rso 50 
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xr 7 72 0 3 ) mm mm s 0 1>\ a** 
7*mm.7 x-xtmt. fwns, m 

fc*timflffl©i**fc£fctttHu W/TN?nfci#©r 
txniML mm* mm, mmmmzm 

[0 0 7 2] 7T772 0 4) £#lt&a$8 0{i, W 

**nfc:»©tt««*i:**y 7 ofesastifcxa 
©^f&iifitesrit&u fl4r©#aiim«fcis«ifi*w 

7X»ti/T©*<HSi:LTfi-»-rs. mmmwi5 

XT 7 7" 2 0 5 ) *»Jtt885 8 0 14, gGlgfcBIJIfc 

[007 3] 7x77206) «RSft;fc3HK£©» 

So 0714, *5SW©»HlS**«)*fci{)©X»it«* 
fr?^©7D-^-h (*©i) T-£S 0 
[0 0 7 4] 7x773 0 l) XSJt«a58 0(4, 

fc£» OfcSlifcflO ©tffMm^ WW»6 0«»6X* 

«hfti^^f^ 7*mmm mmm** 

7x773 0 2) 1«IMI«!!)MKtfll«-P*S»^fctt 
7x7 7 , 3 0 31c$?tU *5T4M*&lcB:*Ty7 
3 0 4lcgtyfSo 

[0 0 7 5] 7x77*3 0 3) ZWmmolts tt 
jR*ft©W»f «fc«^^r-r 7X»©W«im©*IJS 
tiff8£*i:j|ftS(LT, 382b©*©B©fi*E«*tt 
*U Xry7*3 0 6fc«fit«o 7- KIBI©* 

^tc^7 h;baw±©E»w-rs*©^feHri!n?* 

So 

[0 0 7 6] 7nT77'3 0 4) ^»Jt^gP8 0{4, ^ 
^tc^7 h;bSffl±©E«*tf-S-r*. 
xf 7 7 3 0 5 ) n&©fflHHB© 9 -6, EHiWi/h 

©t©*f^fcUTaR-rs. 

XT77'3 0 6) jW^l/TCSfilfiifW-rS, 
[0 0 7 7] 7xT7 7'3 0 7) IWft^l 

7x77^3 0 8) 3HR?nfc3W*a^«9 0fclfl 
( l ) % l ©SGUttWflS : m*~ t LTMtfZ 

ntctg*mf&LT^z>T*xh, mm. gpz-stst 
f-<7<D^m.ifmmmt, ^*u7ot«njnT 



(16) 
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[0 0 7 8] (2) %zommni3&:T*7>Y 

H«WB©»HSHk A*iSn?cB#1il8©#&1i$l£ 

[oo7 9] %mmmim&, ktitntc^m 
ommtw) 7 otia^nfc^it^^mit 

[0080] o) %3(omimmJ5& ■ ttc ± 20 
13© (2) wmtctoxx. n-i»+fcH-^T'fr 

^gtiss-rso 0Utfs H8t^f«t3fc, awn 

£»*fc^&#«?3B®1if8£©SG(J££fmU * 
©4n?SHHfi^fc»^fe©*f&©£flafifc UT^£ 

tS1-^T©B«1t«fcO^Tff^o 
[0 0 8 1] 08fcfe^T^ 0y*£»$©B$A£§« 30 

x»4»©aiSa, b, c tmnzmmMMzxn), 
^ ea^is*©®® b t wffim*<Dmm a , b , c 

X»4»©SHHH«i:**. 

[0082] (4) £4 0«ttinH-V^:£fc, W 

6 1 3 -tf©*^** * TzMz.m%.mm 40 

CtfcWCfctK 6*ffilli*VhS^fc©*mig©Si 

^©fcUTtwstts. 

[0 0 8 3] (5) »5©«ffil«tt»*S:T*Xh 

m. mam, %pm* mmmEmmzz 
nzmwu mmktiu 30 ^nmvittm^o 50 
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ft swift ^im wwtftkEoaHH**** 
nfc4*6nfcfflis'j©**©fli*»^. auBft***. 

[0084] (6) 86©«KSfW£B:*fcx X 

wtft $mm<ommfcM-3<mmmn 
[0085] ecFtcc^^iaifcWiH-rs. 09 

it, *5SB©Sffilfi**i65fc»©X»Jttt*ffdlR© 

7a-^-h (*©2) T?fcS. 

XryTM 0 1) ^ftf^flgttd^SOlCfcW, 

[0 0 8 6] XT7^4 0 2) T*XMMR©SHH£ 

ttttfHR tsasnfcr**. nt«©«p«tai:©ma 

jB*4^6»6ns«[©a*#ai>*j:i:'pfH|-rs. 
[0 0 8 7] BWf $©££(£&* AfcSnfcBWHI 

©iwwiwifc^tu 7 ofc«a?nfc®w»»0i#w* 
mmt**v 7 ot»M?nfc^flH8©i«nii«© 

^ ajg^&£fr£^n3ii©i©^ffiavh£^ 

[0088] $fc, i«s«a©9i, mmm. 

*fc»t«B4«Mai*ttji-r*ct^ *©w«if 
m&mthxzyji^ yw\sX$%jv*9 vMe. 
t&±.x'mwt*$>z> tut* Ex-mmmm % z. t 

;l/^±T*©ffiSt*VjN?^t.©*V affil*©«^t) ©i: 
[0 0 8 9] 7 l T77 , 4 0 3) f^XHtl« 

«Ksm«4if©jiMH«**n^tittji 

U tteR*ftA*SP3 0TlR»LfcX««t>©if©aJ»© 

»aatt*a«'r*^it^i*©^ ->xf 
xr77 , 4 04) sigBa©aa*fTofc»k:5s-3 
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co o 9 o] 7f^4 o 5) mmmv$M&m 

( 7 ) n 7 ©gGign-g^a ; zm&m 8 o 
icfcv^T, «£*TY7^©a^*wjii-si8H^ 

% sow a^^tt &©#iaii < swati* 

*73W£tti:l/C©£«I*i:-f3. 
[0 0 9 1] ttTfcc©35rtt*MlfcKWf *. 0 1 0 

7a-^*-h (*©3) ?&£o 

Xfy75 0l) X*«nfc«««fett©WIIWBi«X 

*Ty75 0 2) hmmimit. XtlZtl 20 

»£ titer** h««©tff^fc©aJ3iiBS&£fr& 

f»6n*n©s**»sci:TfW-ji-r*. 

[0 0 9 2] ■MHR©gfliUfc& X1r$tiTcWmn 

znrz^pmvmmtvzm^t ymbmi.? 30 
T©e«Mv£^fc©^ mm&zfrt>®t>tim 

[o o 9 3] trc, mm®v*>*>mmmt, 

RltgT-fe5o ttfclgUKM^^fe©^ £#7t^ 40 

[0 0 9 4] 7^X7^5 0 3) r**Mm BfMS 
U «BR*ffA*»3 0TWIbfeXS*©i?©aP»© 

A©r7^;i/hfiits^t, Hxnt B&im 
^ti$s& H©afii«fc:is^< % i mammfi 

[009 5] XT77504) SlgPig©3Kj[*fT 50 
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-3 te&tc, te*»©Sii1t «©MH#WI** rV 

7£§t LT©mmiflcfflit-§c 

Xr>y:/5 0 5) SMK«©«^X«*Kj5l^&llRt 

7^75 0 6) «R?nfcX«*a^ijll9 0fcttl 

( 8 ) m 8 aymmnm : tv rtmm 

i#l^;l/?©£Glg£g£f S. Wfctf, ±12© 
(3) T*li. »«?tifc«^il©X»'fli:H-^T-f 

rjWsHRSft-rai^fc^^sE^fe^ (8) m 

[0096] a^^r-f 7X»©s«ie*w-S"rsiB 
mffi&tzmt, m*r<(7<DXgw]'T-<Dm 

© ^r-f7©3Wl^;l/T©«filfi*RS"r*ISIi:x 

r^7©#ttsR*-fco^^ *»?nfc***©;<T 
-f 7fcfctf*^*Hc^3£T©«^£ti-*U 

t u mvmmmmmw lt * tV 7©*» 
^mmfcmzmmm^mom&t trail a 

£t#£ 0 fl*:»*©®fi^fc8T3*flfe» 
tf»©SMKB«fcLTB»c*ffS. B#Ai:B^ai:© 

ia©gi«g i: . b^ b £B& c t (omcommmm 

*tWU *©«*«5yWi:WtS»fc©M©iWU 
^;l/T©i»flia©«IHiei:l2S-r*. 
[0097]® ^ r-f 7©£» l^/I/1f©Sffi(**» 

&-rssK^T^7©#«^*-k:o^T, ia?nfc 

X»+©^r-f7fc*»J*tlUR*-fcJtr*4T©SHtt 

«v^t©*^rw 7©x»^i/T?©affiutfr5*& 

*©SMKB«li:LTii«a*»*. ?e»lc, (W^X««t> 
©HftBt^SSasw^aflttifti: l/TBft c* 

m, m&hm&atmmmt. b^b^b^ 

(J> B®Ai:B«ai:©ia©Sftia) ?:0iJ/Ti«i:S« 
£»fc©fl©£»l/^"C©ffl*fr«©Slitt*fc bT^ 

[0098] a^^r-f7x»©afii«a?*fT3 
^ic, «as*-fcbT«i*?nfci«©ttiSfli«fcWi 
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(ordered labeled tree) fcLTSS* SWfcOl^Tltt 
©0T'&£ O 

[0099] a^^r>r 7X»©a««ai*fi3iBfc, 
*n?n©;£»*©£;<Tw- Tommmmm? 

Dim *fcfctf£/- HOIIttf: LTtettLfcUtttf 

7^;W^t*£©Mft^tt£Jt8hrsc£T\ 
TY7£»©m£(g£fS£?-3o 
[0 10 0] t©a£*r*7:£»©£GU**«rrs 

8{«£i:L-Cg/£T£o 4fc «*1*fR©a«lfl:H\ «• 

y-F©£tu (7<;i/«) ^y-KSx #/-k©&b 
tsfcaflfc-rstofc-fs. 20 

[0101] ±i$©£?lc, jpJffl«*«PJBftX»«BB* 

0, #mf^ttttigP5 0, §9^6 0, ^»it^gP8 0 
fcTW^AfcLTWIRU «&^TY7X«0«t« 

X^B^, 7D? t>— r-fX^CD-ROMtf©^ 30 

[oio2]*^ *«wtt, ±jE©*fltWfci83e«n 

[0 103] 

±&©J:3fcx *»»fcJ:ntf, #Jffl# 
jm»ftX««a%a6ft< T 7*7. HMKJW) 



^*-©H»fc£»T:»©Mlfc«**fT 5 t 
[0®©f§#«J§] 

[0 1 ] *^W©S9*l»»fSfei6©HTfe*o 

[02] *mnewsMLtnhi. 

[S3] *58W©«^^r-f7i«*SiiWr5fe«>OH 

[04] imwv>fe%*Tj7icm<Dmm&mw<Dm 

[0 5] *f^©8te*r-f7:fcaSW7x-X©7D 
-f-lr-hT**. 

[06] «8WO*^7 , -f7XWW7i-X07o 

[07] *56WOS<Kfi***Sfci6©jdlJt«*fi5 
^©70-7^-h (£©1) 
[08] *^OW-»*fcW-^^7JlWHWrtE 

[09] *5IWoaffiUlE**i6Sfci6©X»it«*ff3 
R©7a-?-+-h 0£©2) 
[0 1 0] *%W©SffllSC**i6Sfci6©i»Jt«*fT 
?|&©7P-7*-r- (*©3) T*S5o 
[01 1] *%W©^^T-f7X»©Stt#t«J?7 

10 fg^f^XIXAm S^f^7XtA 

20 mmxtm 

40 %$*T<(7$m®m 
5 o ^aramm mmmm 
e o «am 

7 0 ;**'J 

8 o xatttsm zmvcsm 

9 o ttj^m ^*sb 
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